Can't use curl or wget to query some sites

#1

There seems to be some update or tweak.
Causing me to create a new server and cannot use CURL with some websites.

root@rhino:~# curl -Il -v https://www.cbr.com/squid-game-season-2-filming-start-window/
*   Trying 34.201.177.150:443...
* Connected to www.cbr.com (34.201.177.150) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=cbr.com
*  start date: Jan 30 13:24:33 2023 GMT
*  expire date: Apr 30 13:24:32 2023 GMT
*  subjectAltName: host "www.cbr.com" matched cert's "*.cbr.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x557b1fe3b550)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> HEAD /squid-game-season-2-filming-start-window/ HTTP/2
> Host: www.cbr.com
> user-agent: curl/7.81.0
> accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
* stopped the pause stream!
* Connection #0 to host www.cbr.com left intact
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)

I have tried many methods. Can’t seem to solve this problem. You can test with https://collider.com/ also has problems.

#2

Hello @akoneko47,

We’ll look into the issue and get back to you.

#3

Hello @akoneko47,

I tried running the CURL command you provided on my local machine and on ubuntu server and received the same response. Also, I’m not sure about the -Il command but running the command without -Il flag gave me a proper response.

I hope it helps.

#4

When no flag

root@rhino:~# curl https://www.cbr.com/squid-game-season-2-filming-start-window/
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)

Only -v

root@rhino:~# curl -v https://www.cbr.com/squid-game-season-2-filming-start-window/
*   Trying 34.201.177.150:443...
* Connected to www.cbr.com (34.201.177.150) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=cbr.com
*  start date: Jan 30 13:24:33 2023 GMT
*  expire date: Apr 30 13:24:32 2023 GMT
*  subjectAltName: host "www.cbr.com" matched cert's "*.cbr.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x560cc47d2550)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /squid-game-season-2-filming-start-window/ HTTP/2
> Host: www.cbr.com
> user-agent: curl/7.81.0
> accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
* stopped the pause stream!
* Connection #0 to host www.cbr.com left intact
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)

It’s new server and i tried re create 2 times Ubuntu 22.04, 20.04 at DigitalOcean. Both are problems
The first cause I encountered was php using curl. When I tried curl on command, I got the same problem as above.

#5

Hello @akoneko47,

With the URL you’ve provided I get the same response:

When I tried running the CURL command with cleavr.io/cleavr-slice I got proper response:

So, there must be something with the site you’re performing curl. You may want to try running the curl command with some other URLs for the test purpose.

#6

The problem seems to come from DigitalOcean.
Because I tested at EC2 AWS, no problem found, what do I need to do? Because it’s not just 1-2 websites, but there are a lot of problems, maybe the network system has a problem.

#7

Can you try running the command curl --http1.1 URL?

#8 
root@rhino:~# curl -v https://www.cbr.com/ --http1.1
*   Trying 34.201.177.150:443...
* Connected to www.cbr.com (34.201.177.150) port 443 (#0)
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=cbr.com
*  start date: Jan 30 13:24:33 2023 GMT
*  expire date: Apr 30 13:24:32 2023 GMT
*  subjectAltName: host "www.cbr.com" matched cert's "*.cbr.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/1.1
> Host: www.cbr.com
> User-Agent: curl/7.81.0
> Accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS alert, decode error (562):
* OpenSSL SSL_read: error:0A000126:SSL routines::unexpected eof while reading, errno 0
* Closing connection 0
curl: (56) OpenSSL SSL_read: error:0A000126:SSL routines::unexpected eof while reading, errno 0

ERROR
curl: (56) OpenSSL SSL_read: error:0A000126:SSL routines::unexpected eof while reading, errno 0

#9

Hello @akoneko47,

Did you try running CURL with some other URLs? What was the response? Was it the same error with other URLs as well?

#10

Many websites are normal.
I have tried both HTTP 1.1, 2

#11

@anish

Do you have any solution? Now my team’s work has been forced to stop. because some important systems are not working for 2-3 days
And most importantly, I can’t fix this problem. Because it’s not from my code.

Or do I really have to migrate back to AWS since I recently moved to DO?

#12

Hello @akoneko47,

I spent some time trying to figure out the issue inorder to help you but I couldn’t find anything helpful. May be the site has been configured differently. Give it a last try by re-installing the curl. And if that still doesn’t work may be you’ll have to migrate back to AWS (if it’s working with AWS).

#13

@anish Re-installation or upgrading to the latest version i have tried

Can you contact the DO department as mine seems to be late as it took 1 day to email, if so please.

Why I don’t want to move back because it has a very high cost

#14

And that I will convey It’s not from Curl, so if you try wget or try something else. It can’t be used either.

root@rhino:~# wget https://collider.com/
--2023-02-16 14:50:08--  https://collider.com/
Resolving collider.com (collider.com)... 3.223.39.19
Connecting to collider.com (collider.com)|3.223.39.19|:443... connected.
HTTP request sent, awaiting response... No data received.
Retrying.

--2023-02-16 14:50:10--  (try: 2)  https://collider.com/
Connecting to collider.com (collider.com)|3.223.39.19|:443... connected.
HTTP request sent, awaiting response... No data received.
Retrying.

--2023-02-16 14:50:13--  (try: 3)  https://collider.com/
Connecting to collider.com (collider.com)|3.223.39.19|:443... connected.
HTTP request sent, awaiting response... No data received.
Retrying.

--2023-02-16 14:50:18--  (try: 4)  https://collider.com/
Connecting to collider.com (collider.com)|3.223.39.19|:443... connected.
HTTP request sent, awaiting response... No data received.
Retrying.
#15

Well that confirms it that it has nothing to do with what is installed on the server itself and probably just how things are configured on the target sites as Anish mentioned. Or it could be that DigitalOcean is intercepting those calls and blocking to stop spamming/DDOSing some sites. We are not sure. At this point we are really not able to help with this issue. We can wait to see if someone from the community knows the solution or it might be better to ask this question on StackOverflow.