Daml ledger upload-dar gives GRPCIOTimeout

I have a small x.dar (312K), and I launch an empty Daml Sandbox with daml sandbox, then I get this:

daml ledger upload-dar --host localhost --port 6865 x.dar
Uploading x.dar to localhost:6865
An exception was thrown during the upload-dar command
- GRPCIOTimeout
One reason for this to occur is if the size of DAR file being uploaded exceeds the gRPC maximum message size. The default value for this is 4Mb, but it may be increased when the ledger is (re)started. Please check with your ledger operator.

GRPC should work (I know ), because I can attach to the Sandbox with daml repl or daml ledger navigator.

Also, when I launch the Sandbox via daml sandbox x.dar, it loads the DAR perfectly.

Possibly helpful info: on this VM, I can only run daml test by setting no_proxy=127.0.0.1, otherwise I get
GRPC error (solved here).

Possibly not helpful info: I already checked Daml-helper: GRPCIOTimeout error and GRPCIOTimeout with daml deploy when running within a docker container - #2 by perbergman, this seems to be a different issue.

Richard_Kapolnai:

Possibly helpful info: on this VM, I can only run daml test by setting no_proxy=127.0.0.1 , otherwise I get
GRPC error (solved here).

Did you run your upload-dar command with or without the no_proxy env var set to the same value as your --host option? (Proxy may happen before DNS resolution.)

I think you’ve just led me to a workaround and to a minor bug, thank you!
Apparently daml ledger does not resolve localhost. Manually resolving it resolves my issue:

 daml ledger upload-dar --host 127.0.0.1 --port 6865 x.dar
Uploading x.dar to 127.0.0.1:6865
DAR upload succeeded.

$ daml ledger upload-dar --host localhost --port 6865 x.dar
Uploading x.dar to localhost:6865
An exception was thrown during the upload-dar command
- GRPCIOTimeout
One reason for this to occur is if the size of DAR file being uploaded exceeds the gRPC maximum message size. The default value for this is 4Mb, but it may be increased when the ledger is (re)started. Please check with your ledger operator.

$ host localhost
localhost has address 127.0.0.1

Out of pure curiosity, can you try this?

no_proxy=localhost daml ledger upload-dar --host localhost --port 6865 x.dar

$ no_proxy=localhost daml ledger upload-dar --host localhost --port 6865 x.dar
Uploading x.dar to localhost:6865
DAR upload succeeded.

So it sounds like there isn’t a bug, just a proxy issue?

As a user, I would expect localhost and 127.0.0.1 to behave identically.
We can call it inconvenience instead of a bug , but I doubt I would have beeen able to figure this out on my own.

Afaict they do behave identically? You said at the beginning that you need to set no_proxy=127.0.0.1 for it to work. Now it seems like if you set no_proxy=localhost that works as well. That seems consistent to me?

It looks like the proxy decision is made before DNS resolution (which kinda makes sense), so your no_proxy list has to contain the same hostname you pass to --host. I don’t think we can realistically do anything about how proxies work.

Thank you, these all start to make sense to me. Although I had to set no_proxy=127.0.0.1, otherwise daml test won’t work (I tried, no luck with localhost), but as you wrote I should remember how it is set, and replace all localhost with 127.0.0.1. Although although daml script accepts localhost .
Thanks again for the help!

Let me try to clarify some of the confusion:

1. daml test internally connects to the scenario service via gRPC. It connects via 127.0.0.1 instead of localhost so you always need to set `no_proxy=127.0.0.1z.
2. daml ledger upload-dar uses whatever you supply. If you pass --ledger-host=localhost you need no_proxy=localhost and the other way around.
3. daml script uses the Java gRPC library contrary to the first two which use the (Haskell bindings to the) C++ gRPC library. Proxy handling works differently between the two which is why one can work if the other doesn’t.
   All the proxy handling happens in those underlying libraries so we have little to no control over it unfortunately.

no_proxy is not super standardized AFAIK, but you should be able to give it a list of hostnames. I think the separator is usually a comma, but you may have to check with your actual proxy provider. It’s worth testing, as that’s faster than going through docs:

export no_proxy=localhost,127.0.0.1
daml test
daml ledger upload-dar --host localhost --port 6865 x.dar

I confirm, export no_proxy=localhost,127.0.0.1 worked for me!