Compare commits

..

No commits in common. "master" and "4.6.1" have entirely different histories.

7 changed files with 65 additions and 330 deletions

View File

@ -1,14 +1,5 @@
FROM golang AS builder
WORKDIR /src
RUN git clone https://github.com/tenox7/wrp.git
WORKDIR /src/wrp
RUN go mod download
ARG TARGETARCH
RUN CGO_ENABLED=0 GOOS=linux GOARCH=${TARGETARCH} go build -o /wrp-${TARGETARCH}
FROM chromedp/headless-shell
ARG TARGETARCH
COPY --from=builder /wrp-${TARGETARCH} /wrp
ADD wrp /wrp
ENTRYPOINT ["/wrp"]
ENV PATH="/headless-shell:${PATH}"
LABEL maintainer="as@tenoware.com"

View File

@ -13,11 +13,15 @@ cross:
GOOS=linux GOARCH=arm go build -a -o wrp-arm-linux wrp.go
GOOS=linux GOARCH=arm64 go build -a -o wrp-arm64-linux wrp.go
docker-local:
docker buildx build --platform linux/amd64,linux/arm64 -t tenox7/wrp:latest --load .
docker: wrp
docker build -t tenox7/wrp:latest .
docker-push:
docker buildx build --platform linux/amd64,linux/arm64 -t tenox7/wrp:latest --push .
dockerhub:
docker push tenox7/wrp:latest
gcrio:
docker tag tenox7/wrp:latest gcr.io/tenox7/wrp
docker push gcr.io/tenox7/wrp
clean:
rm -rf wrp-* wrp

119
README.md
View File

@ -1,14 +1,12 @@
# WRP - Web Rendering Proxy
A browser-in-browser "proxy" server that allows to use historical / vintage web browsers on the modern web. It works by rendering a web page in to a GIF or PNG image with clickable imagemap. Optionally by converting modern HTML in to Markdown and back to a simple HTML.
A browser-in-browser "proxy" server that allows to use historical / vintage web browsers on the modern web. It works by rendering a web page in to a GIF or PNG image with clickable imagemap.
![Internet Explorer 1.5 doing Gmail](wrp.png)
## Usage Instructions
### Image Map Mode
* [Download a WRP binary](https://github.com/tenox7/wrp/releases/) run it on a machine that will become your WRP gateway/server. This should be modern hardware and OS. Google Chrome / Chromium Browser is required to be preinstalled. Do not try to run WRP on an old machine like Windows XP or 98.
* [Download a WRP binary](https://github.com/tenox7/wrp/releases/) and run it on a machine that will become your WRP gateway/server. This should be modern hardware, OS and Google Chrome / Chromium Browser is required to be preinstalled. Do not try to run WRP on an old machine like Windows XP or 98.
* Make sure you have disabled firewall or open port WRP is listening on (by default 8080).
* Point your legacy browser to `http://address:port` of the WRP server. Do not set or use it as a "proxy server".
* Type a search string or a full http/https URL and click **Go**.
@ -22,13 +20,6 @@ A browser-in-browser "proxy" server that allows to use historical / vintage web
* Prefer PNG over GIF if your browser supports it. PNG is much faster, whereas GIF requires a lot of additional processing on both client and server to encode/decode. Jpeg encoding is also quite fast.
* GIF images are by default encoded with 216 colors, "web safe" palette. This uses an ultra fast but not very accurate color mapping algorithm. If you want better color representation switch to 256 color mode.
### Simple HTML mode
Also known as **Reader** mode. **NEW!**
* Select image type to `TXT`. WRP will convert the image in to Markdown, then render it back in to a simplified HTML.
* If you want to always start in txt mode specify `-t txt` flag.
## UI explanation
The first unnamed input box is either search (google) or URL starting with http/https
@ -61,67 +52,41 @@ used with PNG and lots of memory on a client side.
**< ^ v >** are arrow keys, typically for navigating a map, buggy.
### UI Customization
### UI Customization
WRP supports customizing it's own UI using HTML Template file. Download [wrp.html](wrp.html) place in the same directory with wrp binary customize it to your liking.
## Docker
https://hub.docker.com/r/tenox7/wrp
```shell
$ docker run -d --rm -p 8080:8080 tenox7/wrp:latest
```
## AWS
It's possible to run WRP on AWS App Runner.
First you need to upload the Docker image to ECR - [Instructions](https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-ecr-image.html).
Create App Runner service using the uploaded image using the AWS Console or CLI.
[AWS Console](https://console.aws.amazon.com/apprunner/home#/create)
```shell
aws apprunner create-service --service-name my-app-runner-service --source-configuration '{
"ImageRepository": {
"ImageIdentifier": "<account_id>.dkr.ecr.<region>.amazonaws.com/wrp:latest",
"ImageRepositoryType": "ECR",
"ImageConfiguration": {"Port": "8000"},
"AutoDeploymentsEnabled": true
}
}' --instance-configuration '{
"Cpu": "1024",
"Memory": "2048",
"InstanceRoleArn": "arn:aws:iam::<account_id>:role/AppRunnerECRAccessRole"
}'
```
## Azure Container Instances
[Azure Console](https://portal.azure.com/#create/Microsoft.ContainerInstances)
CLI:
```shell
$ az container create --resource-group wrp --name wrp --image tenox7/wrp:latest --cpu 1 --memory 2 --ports 80 --protocol tcp --os-type Linux --ip-address Public --command-line '/wrp -l :80 -t png -g 1280x0x256'
$ docker run -d -p 80:8080 tenox7/wrp
```
## Google Cloud Run
```shell
$ gcloud run deploy --platform managed --image=tenox7/wrp:latest --memory=2Gi --args='-t=png','-g=1280x0x256'
$ gcloud run deploy --platform managed --image=gcr.io/tenox7/wrp:latest --memory=2Gi --args='-t=png','-g=1280x0x256'
```
Unfortunately Google Cloud Run forces you to use HTTPS, which likely won't work with old browsers.
Or from [Gcloud Console](https://console.cloud.google.com/run). Use `gcr.io/tenox7/wrp:latest` as container image URL.
Note that unfortunately GCR forces https. Your browser support of encryption protocols and certification authorities will vary.
## Azure Container Instances
```shell
$ az container create --resource-group wrp --name wrp --image gcr.io/tenox7/wrp:latest --cpu 1 --memory 2 --ports 80 --protocol tcp --os-type Linux --ip-address Public --command-line '/wrp -l :80 -t png -g 1280x0x256'
```
Or from the [Azure Console](https://portal.azure.com/#create/Microsoft.ContainerInstances). Use `gcr.io/tenox7/wrp:latest` or `tenox7/wrp:latest` for image name.
Fortunately ACI allows port 80 without encryption.
## Flags
```text
-l listen address:port (default :8080)
-t image type gif, png or jpg (default gif) also txt for simple html/reader mode
-t image type gif, png or jpg (default gif)
-g image geometry, WxHxC, height can be 0 for unlimited (default 1152x600x216)
C (number of colors) is only used for GIF
-q Jpeg image quality, default 80%
@ -129,7 +94,6 @@ Unfortunately Google Cloud Run forces you to use HTTPS, which likely won't work
-d chromedp debug logging (default false)
-n do not free maps and images after use (default false)
-ui html template file (default "wrp.html")
-ua user agent, override the default "headless" agent
-s delay/sleep after page is rendered before screenshot is taken (default 2s)
```
@ -138,48 +102,28 @@ Unfortunately Google Cloud Run forces you to use HTTPS, which likely won't work
* Server/Gateway requires modern hardware and operating system that is supported by [Go language](https://github.com/golang/go/wiki/MinimumRequirements) and Chrome/Chromium Browser, which must be installed.
* Client Browser needs to support `HTML FORMs` and `ISMAP`. Typically [Mosaic 2.0](http://www.ncsa.illinois.edu/enabling/mosaic/versions) would be minimum version for forms. However ISMAP was supported since 0.6B, so if you manually enter url using `?url=...`, you can use the earlier version.
## FAQ
## Troubleshooting
### I can't get it to run
This program does not have a GUI and is run from the command line. After downloading, you may need to enable executable bit on Unix systems, for example:
This program does not have a GUI and is run from the command line. You may need to enable executable bit on Unix systems, for example:
```shell
$ cd ~/Downloads
$ chmod +x wrp-amd64-macos
$ ./wrp-amd64-macos
$ ./wrp-amd64-macos -t png
```
### Websites are blocking headless browsers
This is a well known issue. WRP has some provisions to work around it, but it's a cat and mouse game. The first and
foremost recommendation is to change `User Agent`, so that it doesn't say "headless". Add `-ua="my agent"` to override the default one.
Obtain your regular desktop browser user agent and specify it as the flag. For example
```shell
$ wrp -ua="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36"
```
### Why is WRP called "proxy" when it's not
WRP originally started as true http proxy. However this stopped working because the whole internet is now encrypted thanks to [Let's Encrypt](https://en.wikipedia.org/wiki/Let%27s_Encrypt). Legacy browsers do not support modern SSL/TLS certs as well as [HTTP CONNECT](https://en.wikipedia.org/wiki/HTTP_tunnel#HTTP_CONNECT_method) so this mode had to be disabled.
### Will you support http proxy mode in future?
Some efforts are under way but it's very [difficult](https://en.wikipedia.org/wiki/HTTP_tunnel#HTTP_CONNECT_method) to do it correctly and the priority is rather low.
## History
* Version 1.0 (2014) started as a *cgi-bin* script, adaptation of `webkit2png.py` and `pcidade.py`, [blog post](https://virtuallyfun.com/2014/03/03/surfing-modern-web-with-ancient-browsers/).
* Version 2.0 became a stand alone http-proxy server, supporting both Linux and MacOS, [another post](https://virtuallyfun.com/wordpress/2014/03/11/web-rendering-proxy-update//).
* In 2016 thanks to [Let's Encrypt](https://en.wikipedia.org/wiki/Let%27s_Encrypt) the whole internet migrated to HTTPS/SSL/TLS and WRP largely stopped working. Python code became unmaintainable and there was no easy way to make it work on Windows, even under WSL.
* In 2016 thanks to EFF/Certbot the whole internet migrated to HTTPS/SSL/TLS and WRP largely stopped working. Python code became unmaintainable and there was no easy way to make it work on Windows, even under WSL.
* Version 3.0 (2019) has been rewritten in [Go](https://golang.org/) using [Chromedp](https://github.com/chromedp) as browser-in-browser instead of http-proxy. The initial version was [less than 100 lines of code](https://gist.github.com/tenox7/b0f03c039b0a8b67f6c1bf47e2dd0df0).
* Version 4.0 has been completely refactored to use mouse clicks via imagemap instead parsing a href nodes.
* Version 4.0 has been completely refactored to use mouse clicks via imagemap instead parsing a href nodes.
* Version 4.1 added sending keystrokes in to input boxes. You can now login to Gmail. Also now runs as a Docker container and on Cloud Run/Azure Containers.
* Version 4.5 introduces rendering whole pages in to a single tall image with client scrolling.
* Version 4.6 adds blazing fast gif encoding by [Hill Ma](https://github.com/mahiuchun).
* Version 4.6.3 adds arm64 / aarch64 Docker container support - you can run it on Raspberry PI!
* Version 4.7 add simple html aka reader aka text mode.
## Credits
@ -190,19 +134,8 @@ Some efforts are under way but it's very [difficult](https://en.wikipedia.org/wi
* Thanks to [Hill Ma](https://github.com/mahiuchun) for ultra fast gif encoding algorithm
* Historical Python/Webkit versions and prior art can be seen in [wrp-old](https://github.com/tenox7/wrp-old) repo
## Related
You may also be interested in:
* [VncFox](https://github.com/tenox7/vncfox)
* [Browservice](https://github.com/ttalvitie/browservice)
* [Browsh](https://github.com/browsh-org/browsh)
## Legal Stuff
```text
License: Apache 2.0
Copyright (c) 2013-2024 Antoni Sawicki
Copyright (c) 2019-2024 Google LLC
```
License: Apache 2.0
Copyright (c) 2013-2018 Antoni Sawicki
Copyright (c) 2019-2022 Google LLC

15
go.mod
View File

@ -3,25 +3,18 @@ module github.com/tenox7/wrp
go 1.21.5
require (
github.com/JohannesKaufmann/html-to-markdown v1.6.0
github.com/MaxHalford/halfgone v0.0.0-20171017091812-482157b86ccb
github.com/chromedp/cdproto v0.0.0-20240519224452-66462be74baa
github.com/chromedp/chromedp v0.9.5
github.com/gomarkdown/markdown v0.0.0-20240419095408-642f0ee99ae2
github.com/chromedp/cdproto v0.0.0-20231205062650-00455a960d61
github.com/chromedp/chromedp v0.9.3
github.com/soniakeys/quant v1.0.0
)
require (
github.com/PuerkitoBio/goquery v1.9.2 // indirect
github.com/andybalholm/cascadia v1.3.2 // indirect
github.com/chromedp/sysutil v1.0.0 // indirect
github.com/gobwas/httphead v0.1.0 // indirect
github.com/gobwas/pool v0.2.1 // indirect
github.com/gobwas/ws v1.4.0 // indirect
github.com/gobwas/ws v1.3.1 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/mailru/easyjson v0.7.7 // indirect
github.com/yuin/goldmark v1.7.2 // indirect
golang.org/x/net v0.25.0 // indirect
golang.org/x/sys v0.20.0 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
golang.org/x/sys v0.15.0 // indirect
)

108
go.sum
View File

@ -1,117 +1,29 @@
github.com/JohannesKaufmann/html-to-markdown v1.6.0 h1:04VXMiE50YYfCfLboJCLcgqF5x+rHJnb1ssNmqpLH/k=
github.com/JohannesKaufmann/html-to-markdown v1.6.0/go.mod h1:NUI78lGg/a7vpEJTz/0uOcYMaibytE4BUOQS8k78yPQ=
github.com/MaxHalford/halfgone v0.0.0-20171017091812-482157b86ccb h1:YQ+d0g0P0F/06oDoeEgDHeZCIrnKgLxXcqYOpe8sTuU=
github.com/MaxHalford/halfgone v0.0.0-20171017091812-482157b86ccb/go.mod h1:J86XzS1wgzJPjpQmpriJ+SetP17JSQUd9l+HWQK86jA=
github.com/PuerkitoBio/goquery v1.9.2 h1:4/wZksC3KgkQw7SQgkKotmKljk0M6V8TUvA8Wb4yPeE=
github.com/PuerkitoBio/goquery v1.9.2/go.mod h1:GHPCaP0ODyyxqcNoFGYlAprUFH81NuRPd0GX3Zu2Mvk=
github.com/andybalholm/cascadia v1.3.2 h1:3Xi6Dw5lHF15JtdcmAHD3i1+T8plmv7BQ/nsViSLyss=
github.com/andybalholm/cascadia v1.3.2/go.mod h1:7gtRlve5FxPPgIgX36uWBX58OdBsSS6lUvCFb+h7KvU=
github.com/chromedp/cdproto v0.0.0-20240202021202-6d0b6a386732/go.mod h1:GKljq0VrfU4D5yc+2qA6OVr8pmO/MBbPEWqWQ/oqGEs=
github.com/chromedp/cdproto v0.0.0-20240519224452-66462be74baa h1:T3Ho4BWIkoEoMPCj90W2HIPF/k56qk4JWzTs6JUBxVw=
github.com/chromedp/cdproto v0.0.0-20240519224452-66462be74baa/go.mod h1:GKljq0VrfU4D5yc+2qA6OVr8pmO/MBbPEWqWQ/oqGEs=
github.com/chromedp/chromedp v0.9.5 h1:viASzruPJOiThk7c5bueOUY91jGLJVximoEMGoH93rg=
github.com/chromedp/chromedp v0.9.5/go.mod h1:D4I2qONslauw/C7INoCir1BJkSwBYMyZgx8X276z3+Y=
github.com/chromedp/cdproto v0.0.0-20231011050154-1d073bb38998/go.mod h1:GKljq0VrfU4D5yc+2qA6OVr8pmO/MBbPEWqWQ/oqGEs=
github.com/chromedp/cdproto v0.0.0-20231205062650-00455a960d61 h1:XD280QPATe9jaz20dylKe3vBsNcH1w3mkssGY0lidn8=
github.com/chromedp/cdproto v0.0.0-20231205062650-00455a960d61/go.mod h1:GKljq0VrfU4D5yc+2qA6OVr8pmO/MBbPEWqWQ/oqGEs=
github.com/chromedp/chromedp v0.9.3 h1:Wq58e0dZOdHsxaj9Owmfcf+ibtpYN1N0FWVbaxa/esg=
github.com/chromedp/chromedp v0.9.3/go.mod h1:NipeUkUcuzIdFbBP8eNNvl9upcceOfWzoJn6cRe4ksA=
github.com/chromedp/sysutil v1.0.0 h1:+ZxhTpfpZlmchB58ih/LBHX52ky7w2VhQVKQMucy3Ic=
github.com/chromedp/sysutil v1.0.0/go.mod h1:kgWmDdq8fTzXYcKIBqIYvRRTnYb9aNS9moAV0xufSww=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/gobwas/httphead v0.1.0 h1:exrUm0f4YX0L7EBwZHuCF4GDp8aJfVeBrlLQrs6NqWU=
github.com/gobwas/httphead v0.1.0/go.mod h1:O/RXo79gxV8G+RqlR/otEwx4Q36zl9rqC5u12GKvMCM=
github.com/gobwas/pool v0.2.1 h1:xfeeEhW7pwmX8nuLVlqbzVc7udMDrwetjEv+TZIz1og=
github.com/gobwas/pool v0.2.1/go.mod h1:q8bcK0KcYlCgd9e7WYLm9LpyS+YeLd8JVDW6WezmKEw=
github.com/gobwas/ws v1.3.2/go.mod h1:hRKAFb8wOxFROYNsT1bqfWnhX+b5MFeJM9r2ZSwg/KY=
github.com/gobwas/ws v1.4.0 h1:CTaoG1tojrh4ucGPcoJFiAQUAsEWekEWvLy7GsVNqGs=
github.com/gobwas/ws v1.4.0/go.mod h1:G3gNqMNtPppf5XUz7O4shetPpcZ1VJ7zt18dlUeakrc=
github.com/gomarkdown/markdown v0.0.0-20240419095408-642f0ee99ae2 h1:yEt5djSYb4iNtmV9iJGVday+i4e9u6Mrn5iP64HH5QM=
github.com/gomarkdown/markdown v0.0.0-20240419095408-642f0ee99ae2/go.mod h1:JDGcbDT52eL4fju3sZ4TeHGsQwhG9nbDV21aMyhwPoA=
github.com/gobwas/ws v1.3.0/go.mod h1:hRKAFb8wOxFROYNsT1bqfWnhX+b5MFeJM9r2ZSwg/KY=
github.com/gobwas/ws v1.3.1 h1:Qi34dfLMWJbiKaNbDVzM9x27nZBjmkaW6i4+Ku+pGVU=
github.com/gobwas/ws v1.3.1/go.mod h1:hRKAFb8wOxFROYNsT1bqfWnhX+b5MFeJM9r2ZSwg/KY=
github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY=
github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y=
github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/ledongthuc/pdf v0.0.0-20220302134840-0c2507a12d80 h1:6Yzfa6GP0rIo/kULo2bwGEkFvCePZ3qHDDTC3/J9Swo=
github.com/ledongthuc/pdf v0.0.0-20220302134840-0c2507a12d80/go.mod h1:imJHygn/1yfhB7XSJJKlFZKl/J+dCPAknuiaGOshXAs=
github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0=
github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc=
github.com/orisano/pixelmatch v0.0.0-20220722002657-fb0b55479cde h1:x0TT0RDC7UhAVbbWWBzr41ElhJx5tXPWkIHA2HWPRuw=
github.com/orisano/pixelmatch v0.0.0-20220722002657-fb0b55479cde/go.mod h1:nZgzbfBr3hhjoZnS66nKrHmduYNpc34ny7RK4z5/HM0=
github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/sebdah/goldie/v2 v2.5.3 h1:9ES/mNN+HNUbNWpVAlrzuZ7jE+Nrczbj8uFRjM7624Y=
github.com/sebdah/goldie/v2 v2.5.3/go.mod h1:oZ9fp0+se1eapSRjfYbsV/0Hqhbuu3bJVvKI/NNtssI=
github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAmXWZgo=
github.com/sergi/go-diff v1.3.1 h1:xkr+Oxo4BOQKmkn/B9eMK0g5Kg/983T9DqqPHwYqD+8=
github.com/sergi/go-diff v1.3.1/go.mod h1:aMJSSKb2lpPvRNec0+w3fl7LP9IOFzdc9Pa4NFbPK1I=
github.com/soniakeys/quant v1.0.0 h1:N1um9ktjbkZVcywBVAAYpZYSHxEfJGzshHCxx/DaI0Y=
github.com/soniakeys/quant v1.0.0/go.mod h1:HI1k023QuVbD4H8i9YdfZP2munIHU4QpjsImz6Y6zds=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
github.com/yuin/goldmark v1.7.1 h1:3bajkSilaCbjdKVsKdZjZCLBNPL9pYzrCakKaf4U49U=
github.com/yuin/goldmark v1.7.1/go.mod h1:uzxRWxtg69N339t3louHJ7+O03ezfj6PlliRlaOzY1E=
github.com/yuin/goldmark v1.7.2 h1:NjGd7lO7zrUn/A7eKwn5PEOt4ONYGqpxSEeZuduvgxc=
github.com/yuin/goldmark v1.7.2/go.mod h1:uzxRWxtg69N339t3louHJ7+O03ezfj6PlliRlaOzY1E=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDfU=
golang.org/x/crypto v0.22.0/go.mod h1:vr6Su+7cTlO45qkww3VDJlzDn0ctJvRgYbC2NvXHt+M=
golang.org/x/crypto v0.23.0/go.mod h1:CKFgDieR+mRhux2Lsu27y0fO304Db0wZe70UKqHu0v8=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.9.0/go.mod h1:d48xBJpPfHeWQsugry2m+kC02ZBRGRgulfHnEXEuWns=
golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg=
golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44=
golang.org/x/net v0.24.0/go.mod h1:2Q7sJY5mzlzWjKtYUEXSlBWCdyaioyXzRB2RtU8KVE8=
golang.org/x/net v0.25.0 h1:d/OCCoBEUq33pjydKrGQhw7IlUPI2Oylr+8qLx49kac=
golang.org/x/net v0.25.0/go.mod h1:JkAGAh7GEvH74S6FOH42FLoXpXbE/aqXSrIQjXgsiwM=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.7.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.16.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.19.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.20.0 h1:Od9JTbYCk261bKm4M/mw7AklTlFYIa0bIp9BgSm1S8Y=
golang.org/x/sys v0.20.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
golang.org/x/term v0.7.0/go.mod h1:P32HKFT3hSsZrRxla30E9HqToFYAQPCMs/zFMBUFqPY=
golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo=
golang.org/x/term v0.17.0/go.mod h1:lLRBjIVuehSbZlaOtGMbcMncT+aqLLLmKrsjNrUguwk=
golang.org/x/term v0.19.0/go.mod h1:2CuTdWZ7KHSQwUzKva0cbMg6q2DMI3Mmxp+gKJbskEk=
golang.org/x/term v0.20.0/go.mod h1:8UkIAJTvZgivsXaD6/pH6U9ecQzZ45awqEOzuCvwpFY=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.15.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo=
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
golang.org/x/sys v0.15.0 h1:h48lPFYpsTvQJZF4EKyI4aLHaev3CxivZmv7yZig9pc=
golang.org/x/sys v0.15.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=

113
wrp.go
View File

@ -1,8 +1,8 @@
//
// WRP - Web Rendering Proxy
//
// Copyright (c) 2013-2024 Antoni Sawicki
// Copyright (c) 2019-2024 Google LLC
// Copyright (c) 2013-2018 Antoni Sawicki
// Copyright (c) 2019-2022 Google LLC
//
package main
@ -13,6 +13,7 @@ import (
"embed"
"flag"
"fmt"
"html/template"
"image"
"image/color/palette"
"image/gif"
@ -28,42 +29,30 @@ import (
"net/url"
"os"
"os/signal"
"runtime"
"strconv"
"strings"
"syscall"
"text/template"
"time"
h2m "github.com/JohannesKaufmann/html-to-markdown"
"github.com/JohannesKaufmann/html-to-markdown/plugin"
"github.com/MaxHalford/halfgone"
"github.com/chromedp/cdproto/css"
"github.com/chromedp/cdproto/emulation"
"github.com/chromedp/cdproto/input"
"github.com/chromedp/cdproto/page"
"github.com/chromedp/chromedp"
"github.com/soniakeys/quant/median"
"github.com/yuin/goldmark"
"github.com/yuin/goldmark/ast"
"github.com/yuin/goldmark/extension"
"github.com/yuin/goldmark/parser"
"github.com/yuin/goldmark/text"
"github.com/yuin/goldmark/util"
)
const version = "4.7.0"
const version = "4.6.0"
var (
addr = flag.String("l", ":8080", "Listen address:port, default :8080")
headless = flag.Bool("h", true, "Headless mode / hide browser window (default true)")
noDel = flag.Bool("n", false, "Do not free maps and images after use")
defType = flag.String("t", "gif", "Image type: png|gif|jpg|txt")
defType = flag.String("t", "gif", "Image type: png|gif|jpg")
jpgQual = flag.Int("q", 80, "Jpeg image quality, default 80%")
fgeom = flag.String("g", "1152x600x216", "Geometry: width x height x colors, height can be 0 for unlimited")
htmFnam = flag.String("ui", "wrp.html", "HTML template file for the UI")
delay = flag.Duration("s", 2*time.Second, "Delay/sleep after page is rendered and before screenshot is taken")
userAgent = flag.String("ua", "", "override chrome user agent")
srv http.Server
actx, ctx context.Context
acncl, cncl context.CancelFunc
@ -98,7 +87,6 @@ type uiData struct {
ImgHeight int
MapURL string
PageHeight string
TeXT string
}
// Parameters for HTML print function
@ -110,7 +98,6 @@ type printParams struct {
mapURL string
imgWidth int
imgHeight int
text string
}
// WRP Request
@ -157,7 +144,6 @@ func (rq *wrpReq) parseForm() {
case "png":
case "gif":
case "jpg":
case "txt":
default:
rq.imgType = *defType
}
@ -185,11 +171,10 @@ func (rq *wrpReq) printHTML(p printParams) {
ImgURL: p.imgURL,
MapURL: p.mapURL,
PageHeight: p.pageHeight,
TeXT: p.text,
}
err := htmlTmpl.Execute(rq.w, data)
if err != nil {
fmt.Fprintf(rq.w, "Error: %v", err)
log.Fatal(err)
}
}
@ -222,12 +207,6 @@ func (rq *wrpReq) action() chromedp.Action {
return chromedp.KeyEvent("\u0301")
case ">":
return chromedp.KeyEvent("\u0303")
case "Up":
return chromedp.KeyEvent("\u0308")
case "Dn":
return chromedp.KeyEvent("\u0307")
case "All": // Select all
return chromedp.KeyEvent("a", chromedp.KeyModifiers(input.ModifierCtrl))
}
}
// Keys
@ -236,7 +215,7 @@ func (rq *wrpReq) action() chromedp.Action {
return chromedp.KeyEvent(rq.keys)
}
// Navigate to URL
log.Printf("%s Processing Navigate Request for %s\n", rq.r.RemoteAddr, rq.url)
log.Printf("%s Processing Capture Request for %s\n", rq.r.RemoteAddr, rq.url)
return chromedp.Navigate(rq.url)
}
@ -317,7 +296,8 @@ func gifPalette(i image.Image, n int64) image.Image {
return i
}
func (rq *wrpReq) captureImage() {
// Capture currently rendered web page to an image and fake ISMAP
func (rq *wrpReq) capture() {
var styles []*css.ComputedStyleProperty
var r, g, b int
var h int64
@ -334,12 +314,12 @@ func (rq *wrpReq) captureImage() {
return nil
}),
)
log.Printf("%s Landed on: %s, Height: %v\n", rq.r.RemoteAddr, rq.url, h)
for _, style := range styles {
if style.Name == "background-color" {
fmt.Sscanf(style.Value, "rgb(%d,%d,%d)", &r, &g, &b)
}
}
log.Printf("%s Landed on: %s, Height: %v\n", rq.r.RemoteAddr, rq.url, h)
height := int64(float64(rq.height) / rq.zoom)
if rq.height == 0 && h > 0 {
height = h + 30
@ -418,63 +398,6 @@ func (rq *wrpReq) captureImage() {
log.Printf("%s Done with capture for %s\n", rq.r.RemoteAddr, rq.url)
}
func asciify(s []byte) []byte {
a := make([]byte, len(s))
for i := 0; i < len(s); i++ {
if s[i] > 127 {
a[i] = '.'
continue
}
a[i] = s[i]
}
return a
}
type astTransformer struct{}
func (t *astTransformer) Transform(node *ast.Document, reader text.Reader, pc parser.Context) {
ast.Walk(node, func(n ast.Node, entering bool) (ast.WalkStatus, error) {
if link, ok := n.(*ast.Link); ok && entering {
link.Destination = append([]byte("?t=txt&url="), link.Destination...)
}
if _, ok := n.(*ast.Image); ok && entering {
// TODO: perhaps instead of deleting images convert them to links
// smaller images or ascii? https://github.com/TheZoraiz/ascii-image-converter
n.Parent().RemoveChildren(n)
}
return ast.WalkContinue, nil
})
}
func (rq *wrpReq) captureMarkdown() {
log.Printf("Processing Markdown conversion request for %v", rq.url)
// TODO: bug - DomainFromURL always prefixes with http:// instead of https
// this causes issues on some websites, fix or write a smarter DomainFromURL
c := h2m.NewConverter(h2m.DomainFromURL(rq.url), true, nil)
c.Use(plugin.GitHubFlavored())
md, err := c.ConvertURL(rq.url) // We could also get inner html from chromedp
if err != nil {
http.Error(rq.w, err.Error(), http.StatusInternalServerError)
return
}
log.Printf("Got %v bytes md from %v", len(md), rq.url)
gm := goldmark.New(
goldmark.WithExtensions(extension.GFM),
goldmark.WithParserOptions(parser.WithASTTransformers(util.Prioritized(&astTransformer{}, 100))),
)
var ht bytes.Buffer
err = gm.Convert([]byte(md), &ht)
if err != nil {
http.Error(rq.w, err.Error(), http.StatusInternalServerError)
return
}
log.Printf("Rendered %v bytes html for %v", len(ht.String()), rq.url)
rq.printHTML(printParams{
text: string(asciify([]byte(ht.String()))),
bgColor: "#FFFFFF",
})
}
// Process HTTP requests to WRP '/' url
func pageServer(w http.ResponseWriter, r *http.Request) {
log.Printf("%s Page Request for %s [%+v]\n", r.RemoteAddr, r.URL.Path, r.URL.RawQuery)
@ -488,11 +411,7 @@ func pageServer(w http.ResponseWriter, r *http.Request) {
return
}
rq.navigate() // TODO: if error from navigate do not capture
if rq.imgType == "txt" {
rq.captureMarkdown()
return
}
rq.captureImage()
rq.capture()
}
// Process HTTP requests to ISMAP '/map/' url
@ -521,7 +440,7 @@ func mapServer(w http.ResponseWriter, r *http.Request) {
return
}
rq.navigate() // TODO: if error from navigate do not capture
rq.captureImage()
rq.capture()
}
// Process HTTP requests for images '/img/' url
@ -628,9 +547,8 @@ func printIPs(b string) {
// Main
func main() {
var err error
log.SetFlags(log.LstdFlags | log.Lshortfile)
flag.Parse()
log.Printf("Web Rendering Proxy Version %s (%v)\n", version, runtime.GOARCH)
log.Printf("Web Rendering Proxy Version %s\n", version)
log.Printf("Args: %q", os.Args)
if len(os.Getenv("PORT")) > 0 {
*addr = ":" + os.Getenv(("PORT"))
@ -644,12 +562,7 @@ func main() {
opts := append(chromedp.DefaultExecAllocatorOptions[:],
chromedp.Flag("headless", *headless),
chromedp.Flag("hide-scrollbars", false),
chromedp.Flag("enable-automation", false),
chromedp.Flag("disable-blink-features", "AutomationControlled"),
)
if *userAgent != "" {
opts = append(opts, chromedp.UserAgent(*userAgent))
}
actx, acncl = chromedp.NewExecAllocator(context.Background(), opts...)
defer acncl()
ctx, cncl = chromedp.NewContext(actx)

View File

@ -6,12 +6,9 @@
<FORM ACTION="/" METHOD="POST">
<INPUT TYPE="TEXT" NAME="url" VALUE="{{.URL}}" SIZE="20">
<INPUT TYPE="SUBMIT" VALUE="Go">
{{ if ne .ImgType "txt" }}
<INPUT TYPE="SUBMIT" NAME="Fn" VALUE="Bk">
<INPUT TYPE="SUBMIT" NAME="Fn" VALUE="St">
<INPUT TYPE="SUBMIT" NAME="Fn" VALUE="Re">
<INPUT TYPE="SUBMIT" NAME="Fn" VALUE="Up">
<INPUT TYPE="SUBMIT" NAME="Fn" VALUE="Dn">
W <INPUT TYPE="TEXT" NAME="w" VALUE="{{.Width}}" SIZE="4">
H <INPUT TYPE="TEXT" NAME="h" VALUE="{{.Height}}" SIZE="4">
Z <SELECT NAME="z">
@ -23,14 +20,11 @@
<OPTION VALUE="1.2" {{ if eq .Zoom 1.2}}SELECTED{{end}}>1.2 x</OPTION>
<OPTION VALUE="1.3" {{ if eq .Zoom 1.3}}SELECTED{{end}}>1.3 x</OPTION>
</SELECT>
{{ end }}
T <SELECT NAME="t">
<OPTION VALUE="png" {{ if eq .ImgType "png"}}SELECTED{{end}}>PNG</OPTION>
<OPTION VALUE="gif" {{ if eq .ImgType "gif"}}SELECTED{{end}}>GIF</OPTION>
<OPTION VALUE="jpg" {{ if eq .ImgType "jpg"}}SELECTED{{end}}>JPG</OPTION>
<OPTION VALUE="txt" {{ if eq .ImgType "txt"}}SELECTED{{end}}>TXT</OPTION>
</SELECT>
{{ if ne .ImgType "txt" }}
C <SELECT NAME="c">
<OPTION VALUE="256" {{ if eq .NColors 256}}SELECTED{{end}}>256</OPTION>
<OPTION VALUE="216" {{ if eq .NColors 216}}SELECTED{{end}}>216</OPTION>
@ -41,13 +35,11 @@
</SELECT>
K <INPUT TYPE="TEXT" NAME="k" VALUE="" SIZE="4">
<INPUT TYPE="SUBMIT" NAME="Fn" VALUE="Bs">
<INPUT TYPE="SUBMIT" NAME="Fn" VALUE="Rt">
<INPUT TYPE="SUBMIT" NAME="Fn" VALUE="All"><!--
<INPUT TYPE="SUBMIT" NAME="Fn" VALUE="Rt"><!--
<INPUT TYPE="SUBMIT" NAME="Fn" VALUE="&lt;">
<INPUT TYPE="SUBMIT" NAME="Fn" VALUE="^">
<INPUT TYPE="SUBMIT" NAME="Fn" VALUE="v">
<INPUT TYPE="SUBMIT" NAME="Fn" VALUE="&gt;" SIZE="1">-->
{{ end }}
</FORM>
<BR>
{{if .ImgURL}}
@ -56,14 +48,11 @@
</A>
<P>
{{end}}
{{.TeXT}}
<FONT SIZE="-2">
<A HREF="/?url=https://github.com/tenox7/wrp/&w={{.Width}}&h={{.Height}}&s={{printf "%.1f" .Zoom}}&c={{.NColors}}&t={{.ImgType}}">Web Rendering Proxy {{.Version}}</A> |
<A HREF="/shutdown/">Shutdown WRP</A> |
{{ if ne .ImgType "txt" }}
<A HREF="/">Page Height: {{.PageHeight}}</A> |
<A HREF="/">Img Size: {{.ImgSize}}</A>
{{end}}
</FONT>
</FONT>
</BODY>
</HTML>
</HTML>