Discussions/App Development/"No access to: class..." error when attempting to start a domain in CantonForum ↗

"No access to: class..." error when attempting to start a domain in Canton

App Development34 posts1,131 views22 likesLast activity Sep 2021
MA
MatheusOP
Sep 2021

Hi.

I’m working on a canton deployment using canton-enterprise-0.27.0 within a docker container and I’m facing some issues when attempting to start a domain.

Canton starts as expected, but when running domain start the following error comes up and Canton crashes:

ERROR c.d.c.e.EnterpriseEnvironment - A fatal error has occurred in canton-env-execution-context. Terminating immediately.
java.lang.BootstrapMethodError: java.lang.IllegalAccessError: Class 'scala.util.Random' no access to: class 'scala.collection.immutable.LazyList$State'
        at scala.util.Random.alphanumeric(Random.scala:252)
        at com.digitalasset.canton.crypto.PseudoRandom$.randomAlphaNumericString(Random.scala:55)
        at com.digitalasset.canton.identity.IdentityElementId$.generate(IdentityTransaction.scala:354)
        at com.digitalasset.canton.identity.IdentityTransaction$.createAdd(IdentityTransaction.scala:966)
        at com.digitalasset.canton.environment.CantonNodeBootstrapBase.auth(CantonNodeBootstrap.scala:257)
        at com.digitalasset.canton.domain.DomainBootstrap.$anonfun$autoInitializeIdentity$10(Domain.scala:141)
        at com.digitalasset.canton.domain.DomainBootstrap$$Lambda$2468/00000000E834BCB0.apply(Unknown Source)
        at cats.data.EitherT.$anonfun$flatMap$1(EitherT.scala:391)
        at cats.data.EitherT$$Lambda$2333/000000007FF8F0F0.apply(Unknown Source)
        at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:434)
        at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1413)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:300)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1067)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1703)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:168)
Caused by: java.lang.IllegalAccessError: Class 'scala.util.Random' no access to: class 'scala.collection.immutable.LazyList$State'
        at java.lang.invoke.MethodHandle.sendResolveMethodHandle(MethodHandle.java:1084)
        at java.lang.invoke.MethodHandle.getCPMethodHandleAt(Native Method)
        at java.lang.invoke.MethodHandle.getAdditionalBsmArg(MethodHandle.java:867)
        at java.lang.invoke.MethodHandle.resolveInvokeDynamic(MethodHandle.java:946)
        ... 15 common frames omitted
Caused by: java.lang.IllegalAccessException: Class 'scala.util.Random' no access to: class 'scala.collection.immutable.LazyList$State'
        at java.lang.invoke.MethodHandles$Lookup.checkClassAccess(MethodHandles.java:449)
        at java.lang.invoke.MethodHandles$Lookup.accessCheckArgRetTypes(MethodHandles.java:686)
        at java.lang.invoke.MethodHandle.sendResolveMethodHandle(MethodHandle.java:1058)
        ... 18 common frames omitted

Apparently it is not able to reach one of the classes but I wonder why that might be. Is there a dependency missing?

In the image we have the Jar and a config file, is anything else needed?

Thanks!
Matheus

RA
Ratko_Veprek
Sep 2021

Hi @Matheus

Can you help us reproduce the error? I tried it on my (osx) laptop, but it seems to work fine:

ratkoveprekcmg8wl:~ ravep$ docker run -it  digitalasset-canton-enterprise-docker.jfrog.io/digitalasset/canton-enterprise:0.27.0
Compiling /canton/(console)
   _____            _
  / ____|          | |
 | |     __ _ _ __ | |_ ___  _ __
 | |    / _` | '_ \| __/ _ \| '_ \
 | |___| (_| | | | | || (_) | | | |
  \_____\__,_|_| |_|\__\___/|_| |_|

  Welcome to Canton!
  Type `help` to get started. `exit` to leave.


@ mydomain.start()

@ mydomain.health.status
res1: com.digitalasset.canton.health.admin.data.NodeStatus[com.digitalasset.canton.health.admin.data.DomainStatus] = Domain id: mydomain::1220678b58f0c61e5adff797abd83553d4755bccbaaf83b2a8fe8b9f27e6b3599ec1
Uptime: 13.11351s
Ports:
        public: 5018
        admin: 5019
Connected Participants: None
Sequencer: Some(SequencerHealthStatus(isActive = true))

Thanks!
Ratko

MA
Matheus
Sep 2021

Hi @Ratko_Veprek
We are not using the docker image available in DA’s jFrog, but creating one using the artifacts in the canton enterprise zip.
Apparently I was using the wrong java version(8), switching to java 11 I only get this while running bin/canton -c participant.conf --debug:

INFO  c.d.c.CantonEnterpriseApp$ - Starting Canton version 0.27.0
INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
INFO  c.d.c.e.EnterpriseEnvironment - Creating ForkJoinPool with parallelism = 2 (instead of 1) to avoid starvation.
INFO  a.e.s.Slf4jLogger - Slf4jLogger started
INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
INFO  c.d.c.e.EnterpriseEnvironment tid:0ac02f867df67e90f90b81c34b390402 - Manual start requested.

then the terminal seems to finish the process without any errors. Is this expected? how can I check if it is running or not in this variant?

Thanks again.
Matheus

RA
Ratko_Veprek
Sep 2021

Hi @Matheus

Please note that we test against Java 11 and require you to use 11 or higher (compatibility matrix in https://github.com/digital-asset/canton/releases/tag/v0.27.0). This is due to a bug in the JRE libraries pre Java11 which leads to premature thread termination of the ForkJoinPool.

To your question:

  1. If you start ./bin/canton -c participant.conf, it should start in interactive mode and you should get to the console prompt.
  2. are you running ./bin/canton -c participant.conf within docker? This way you are starting the interactive mode and this only works in docker if you run it with docker run -it ....
  3. Alternatively for docker, you should invoke ./bin/canton daemon -c participant.conf. You can then use a remote console to connect to your node.
  4. if you are not running it in docker, then please check the log file log/canton.log what the last sign of life was.
  5. When running in docker, you might want to use --log-profile=container in order to turn on logging to STDOUT as JSON. The --log-... command line arguments allow you to tune the logging output.

Please let me know what you find.

Best,
Ratko

MA
Matheus
Sep 2021

Hi @Ratko_Veprek

Thanks for the reply. Starting with the command mentioned in 0. this is the result after some seconds:

sh-4.4$ ./bin/canton -c examples/01-simple-topology/simple-topology.conf 
sh-4.4$
  1. We’re running this on a Kubernetes pod that runs docker images, unfortunately docker is not present in the image for us to run it from within it.

When providing the canton-enterprise:0.27.0 image from Digital Asset’s repository “directly” to be run in the pod this error pops up in the terminal:

standard_init_linux.go:219: exec user process caused: exec format error
  1. This is the content in log/canton.log for the variant mentioned in 0.:
2021-09-08 10:41:34,313 [main] INFO  c.d.canton.CantonEnterpriseApp$ - Starting Canton version 0.27.0
2021-09-08 10:41:35,098 [main] INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
2021-09-08 10:41:35,101 [main] INFO  c.d.c.e.EnterpriseEnvironment - Creating ForkJoinPool with parallelism = 2 (instead of 1) to avoid starvation.
2021-09-08 10:41:35,369 [canton-env-execution-context-19] INFO  akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2021-09-08 10:41:35,510 [main] INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
2021-09-08 10:41:35,699 [main] INFO  c.d.c.e.EnterpriseEnvironment tid:a778f4e96ce78237234707d1ee46ef79 - Manual start requested.
  1. running it with --log-profile=container results in:
sh-4.4$ bin/canton -c examples/01-simple-topology/simple-topology.conf --log-profile=container
INFO  c.d.c.CantonEnterpriseApp$ - Starting Canton version 0.27.0
INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
INFO  c.d.c.e.EnterpriseEnvironment - Creating ForkJoinPool with parallelism = 2 (instead of 1) to avoid starvation.
INFO  a.e.s.Slf4jLogger - Slf4jLogger started
INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
INFO  c.d.c.e.EnterpriseEnvironment tid:30bda102eacae04df462e42593ba4bfa - Manual start requested.
sh-4.4$

I believe it’s also worth mentioning that we are running on s390x architecture, and we are using an openjdk 11 image for building “our” canton-enterprise image.

Cheers,
Matheus

RA
Ratko_Veprek
Sep 2021

Does by any chance starting canton with

bin/canton -v --no-tty -c examples/01-simple-topology/simple-topology.conf

make any difference?

Based on the output I’d say it fails to initialise the “interactive console”.

For prod ops, I would anyway recommend to run the console separate from the process. In that case you would start with

./bin/canton daemon -v -c examples/01-simple-topology/simple-topology.conf --bootstrap=examples/01-simple-topology/simple-ping.canton

If the latter works, then it’s the interactive console for sure.

MA
Matheus
Sep 2021

the first command didn’t provide any changes to the behaviour.

The second however rendered an exeption:

ERROR c.d.c.ServerRunner - An internal error occurred while executing script.
java.lang.ExceptionInInitializerError: null
        at ammonite.Main$.apply$default$5(Main.scala:68)
        at com.digitalasset.canton.console.HeadlessConsole$.createDefaultAmmoniteOptions(HeadlessConsole.scala:59)
        at com.digitalasset.canton.console.HeadlessConsole$.$anonfun$apply$1(HeadlessConsole.scala:46)
        at better.files.Dispose.apply(Dispose.scala:81)
        at com.digitalasset.canton.console.HeadlessConsole$.apply(HeadlessConsole.scala:45)
        at com.digitalasset.canton.ConsoleScriptRunner$.run(Runner.scala:169)
        at com.digitalasset.canton.ServerRunner.startWithBootstrap$1(Runner.scala:51)
        at com.digitalasset.canton.ServerRunner.$anonfun$run$3(Runner.scala:53)
        at com.digitalasset.canton.ServerRunner.$anonfun$run$3$adapted(Runner.scala:53)
        at scala.Option.fold(Option.scala:263)
        at com.digitalasset.canton.ServerRunner.run(Runner.scala:53)
        at com.digitalasset.canton.CantonAppDriver.delayedEndpoint$com$digitalasset$canton$CantonAppDriver$1(CantonAppDriver.scala:145)
        at com.digitalasset.canton.CantonAppDriver$delayedInit$body.apply(CantonAppDriver.scala:28)
        at scala.Function0.apply$mcV$sp(Function0.scala:39)
        at scala.Function0.apply$mcV$sp$(Function0.scala:39)
        at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
        at scala.App.$anonfun$main$1(App.scala:76)
        at scala.App.$anonfun$main$1$adapted(App.scala:76)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:919)
        at scala.App.main(App.scala:76)
        at scala.App.main$(App.scala:74)
        at com.digitalasset.canton.CantonAppDriver.main(CantonAppDriver.scala:28)
        at com.digitalasset.canton.CantonEnterpriseApp.main(CantonEnterpriseApp.scala)
Caused by: java.lang.IllegalArgumentException: requirement failed: ? is not an absolute path
        at scala.Predef$.require(Predef.scala:338)
        at os.Path.<init>(Path.scala:430)
        at os.Path$.apply(Path.scala:388)
        at os.package$.<clinit>(package.scala:19)
        ... 25 common frames omitted
ERROR c.d.c.ServerRunner - Unexpected error while running server: null
DA
davidpadbury
Sep 2021

Oh dear. I suspect that you may have hit a bug with our script support on this architecture which may require a patch to fix.

I assume running the above without the --bootstrap-script is successful? (but with the problem that we can’t connect participants to domains)

I’ll check in with the team in the meantime.

RA
Ratko_Veprek
Sep 2021

So, I’ve tried out to run Canton on S390 by using the LinuxOne Community Cloud of IBM:

[linux1@cantontest canton-enterprise-0.27.0]$ cat /proc/cpuinfo  | grep IBM/S390
vendor_id       : IBM/S390
[linux1@cantontest canton-enterprise-0.27.0]$ ./bin/canton -c examples/01-simple-topology/simple-topology.conf
   _____            _
  / ____|          | |
 | |     __ _ _ __ | |_ ___  _ __
 | |    / _` | '_ \| __/ _ \| '_ \
 | |___| (_| | | | | || (_) | | | |
  \_____\__,_|_| |_|\__\___/|_| |_|

  Welcome to Canton!
  Type `help` to get started. `exit` to leave.


@ nodes.local.start()


@ participants.all.domains.connect_local(mydomain)


@ participant1.health.ping(participant2)
res2: concurrent.duration.Duration = 2757 milliseconds

@ participant1.health.ping(participant2)
res3: concurrent.duration.Duration = 431 milliseconds

It looks healthy. So from a first glance, it does look to work and I don’t observe any anormalities.

Now, going back to the exceptions you’ve reported. The exception

at ammonite.Main$.apply$default$5(Main.scala:68)

says that this line of code here is throwing:

github.com

com-lihaoyi/os-lib/blob/5dce68dc6588370b7c941b59f2faaa1fa4aaae90/os/src-jvm/package.scala#L24

    
      
    
          
  1. }
    2. 
          
  2. 
          
  3. /**
    4. 
          
  4.  * The user's home directory
    5. 
          
  5.  */
    6. 
          
  6. val home: Path = Path(System.getProperty("user.home"))
    7. 
          
  7. 
          
  8. /**
    9. 
          
  9.  * The current working directory for this process.
    10. 
          
  10.  */
    11. 
          
  11. val pwd: Path = os.Path(java.nio.file.Paths.get(".").toAbsolutePath)
    12. 
          
  12. 
          
  13. val up: RelPath = RelPath.up
    14. 
          
  14. 
          
  15. val rel: RelPath = RelPath.rel
    16. 
          
  16. 
          
  17. val sub: SubPath = SubPath.sub
    18. 
          
  18. /**
    19. 
          
  19.   * Extractor to let you easily pattern match on [[os.Path]]s. Lets you do
    20. 
          
  20.   *
    21. 
          
  21.   * {{{
    22.

Which brings me to the question: do you run this in a way so that the process does not have any access to the disk?

The ammonite console we are embedding in Canton will cache a few files on your file system and it seems that a trivial operation such as determining the absolute path of the current directory fails with an exception.

Can you confirm that the file system is writable and discoverable to the jvm process?

RA
Ratko_Veprek
Sep 2021

So, I can now reproduce more or less the issue you have by running Canton in a read-only docker setup:

docker run --read-only --rm -it -v $PWD/log:/canton/log digitalasset/canton-enterprise:dbg -c log/nope.conf,examples/01-simple-topology/simple-topology.conf --log-level-stdout=TRACE --log-level-root=TRACE

Ammonite requires the ability to create temporary files in order to compile the scripts that we pass as bootstrap or the commands that we type into the console. This is very deep down in the library (it uses java.nio.file.Files.createTempFile) and we can not work around it. Even if I turn off caching of the compiled artefacts (which is the first error you have been hitting), the issue will persist.

However, @davidpadbury’s workaround from above is actually fine. So I guess you have the following options:

  1. Start Canton in daemon mode: ./bin/canton daemon -c <config> but do not pass a --bootstrap=... file and do not start it in interactive mode.
  2. Make the system less locked down and give Canton a home where it can create temporary files.

If you omit the --bootstrap option, you can still use a remote console that has access to the process admin-api in order to run scripts against the node in daemon mode. Have a look at examples/03-advanced-configuration/remote for how to configure a remote participant.

Cheers,
Ratko

MA
Matheus
Sep 2021

I removed any access control to the file system but the behavior didn’t change, so I went with option 1.

Then canton seems to launch in daemon but how do I actually run the script from the participant node if it runs on an identical setup, and also has to be run in daemon mode?

Thanks,
Matheus

DA
davidpadbury
Sep 2021

Hey @Matheus,

I think in short to run the canton console it needs a writable file system. The process running the console can be remote from the servers themselves, but needs a writable file system for the period it’s running commands somewhere.

Could you share a little more about your setup? I’m optimistic that maybe we could work around this by using a ram disk of some form as this doesn’t actually need to write much, so perhaps is achievable with tmpfs via docker or a plain mount.

David.

MA
Matheus
Sep 2021

Hi @davidpadbury
We’re running it in a Kubernetes cluster, the image we build is run in a container within a pod.

There is the possibility of mounting some persistent volumes to which the application can write to, do you know the path it attempts to write to? I’d be happy to give it a try.

Thanks in advance,
Matheus

DA
davidpadbury
Sep 2021

Hey @Matheus,

Okay - thanks for the info.

So I’d suggest creating a writable volume and mounting that into your pod+container. Looks like you could also just create a tmpfs volume easily in k8 that should be sufficient for these purposes.

The tricky bit is that the chunk of code that is failing is using a java API to create temporary files and directories. I think we can actually control where those are based by setting the java system property java.io.tmpdir. We can do this when launching canton either directly or with the docker container by setting the environment variable:

JAVA_OPTS="-Djava.io.tmpdir=/path/to/your/writeable/mount"

Let me know how that works for you. Good luck!

David.

MA
Matheus
Sep 2021

Hi @davidpadbury ,
Unfortunately not much difference, in the logs below the mounted volume is in /home/app/tmp (canton resides in /home/app), Canton didn’t launch in interactive mode, but from the terminal I was able to create files under /home/app/tmp. Also, the JAVA_OPTS variable is set:

sh-4.4$ pwd
/home/app
sh-4.4$ ls
LICENSE.txt  bin   daml  demo        di-execution-integration.dar  di-execution-test.dar  lib  start-demo-win.cmd  start.sh                   tmp
README.md    conf  dars  deployment  di-execution-main.dar         examples               log  start-demo.command  third-party-licenses.html
sh-4.4$ echo $JAVA_OPTS
-Djava.io.tmpdir=/home/app/tmp
sh-4.4$ bin/canton -c examples/01-simple-topology/simple-topology.conf 
sh-4.4$ mkdir tmp/folder
sh-4.4$ echo "testfile" > tmp/folder/test.txt
sh-4.4$ cat tmp/folder/test.txt 
testfile

Is there anything I missed?

Thanks once again,
Matheus

DA
davidpadbury
Sep 2021

Oh dear. What output did you get when canton launched and could you share the log file?

Actually on that thought… When launching canton that way it will try writing a log file to log/canton.log which is maybe problematic given that location won’t be writable. Can you try adding --log-file-name=$PWD/tmp/canton.log just to make sure that ends up somewhere writable too?

MA
Matheus
Sep 2021

That is the weird part to me, there is no output it executes and “finishes” after a second or two.

I ran it again with --log-file-name=$PWD/tmp/canton.log :

sh-4.4$  bin/canton -c examples/01-simple-topology/simple-topology.conf --log-file-name=$PWD/tmp/canton.log
sh-4.4$ cat $PWD/tmp/canton.log
2021-09-10 17:47:22,411 [main] INFO  c.d.canton.CantonEnterpriseApp$ - Starting Canton version 0.27.0
2021-09-10 17:47:23,141 [main] INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
2021-09-10 17:47:23,143 [main] INFO  c.d.c.e.EnterpriseEnvironment - Creating ForkJoinPool with parallelism = 2 (instead of 1) to avoid starvation.
2021-09-10 17:47:23,343 [canton-env-execution-context-19] INFO  akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2021-09-10 17:47:23,392 [main] INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
2021-09-10 17:47:23,485 [main] INFO  c.d.c.e.EnterpriseEnvironment tid:cdc2b1bbc3fa55e17a0aec41db02fa97 - Manual start requested.
DA
davidpadbury
Sep 2021

Great… I assume the process is exiting unsuccessfully? Could you echo $? immediately after canton finishes?

Could you also add --debug to see if we can capture anything useful in the log?

Sorry about this :grimacing:

MA
Matheus
Sep 2021

No need to apologise, I am really greatful for the support :slight_smile:

the output running with --debug and echo $? afterwards:

sh-4.4$  bin/canton -c examples/01-simple-topology/simple-topology.conf --log-file-name=$PWD/tmp/canton.log --debug
INFO  c.d.c.CantonEnterpriseApp$ - Starting Canton version 0.27.0
INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
INFO  c.d.c.e.EnterpriseEnvironment - Creating ForkJoinPool with parallelism = 2 (instead of 1) to avoid starvation.
INFO  a.e.s.Slf4jLogger - Slf4jLogger started
INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
INFO  c.d.c.e.EnterpriseEnvironment tid:2e24416c2789880eb605da153f34cc21 - Manual start requested.
sh-4.4$ echo $?
1
DA
davidpadbury
Sep 2021

And nothing more in the logs?

MA
Matheus
Sep 2021

here the canton.log:

sh-4.4$ cat $PWD/tmp/canton.log 
2021-09-10 17:47:22,411 [main] INFO  c.d.canton.CantonEnterpriseApp$ - Starting Canton version 0.27.0
2021-09-10 17:47:23,141 [main] INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
2021-09-10 17:47:23,143 [main] INFO  c.d.c.e.EnterpriseEnvironment - Creating ForkJoinPool with parallelism = 2 (instead of 1) to avoid starvation.
2021-09-10 17:47:23,343 [canton-env-execution-context-19] INFO  akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2021-09-10 17:47:23,392 [main] INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
2021-09-10 17:47:23,485 [main] INFO  c.d.c.e.EnterpriseEnvironment tid:cdc2b1bbc3fa55e17a0aec41db02fa97 - Manual start requested.
2021-09-10 17:54:50,887 [main] INFO  c.d.canton.CantonEnterpriseApp$ - Starting Canton version 0.27.0
2021-09-10 17:54:51,895 [main] INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
2021-09-10 17:54:51,898 [main] INFO  c.d.c.e.EnterpriseEnvironment - Creating ForkJoinPool with parallelism = 2 (instead of 1) to avoid starvation.
2021-09-10 17:54:52,181 [canton-env-execution-context-19] INFO  akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2021-09-10 17:54:52,249 [main] INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
2021-09-10 17:54:52,325 [main] DEBUG io.grpc.NameResolverRegistry - Service loader found io.grpc.internal.DnsNameResolverProvider@4d192aef
2021-09-10 17:54:52,355 [main] INFO  c.d.c.e.EnterpriseEnvironment tid:2e24416c2789880eb605da153f34cc21 - Manual start requested.
2021-09-10 18:01:00,156 [main] INFO  c.d.canton.CantonEnterpriseApp$ - Starting Canton version 0.27.0
2021-09-10 18:01:00,813 [main] INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
2021-09-10 18:01:00,816 [main] INFO  c.d.c.e.EnterpriseEnvironment - Creating ForkJoinPool with parallelism = 2 (instead of 1) to avoid starvation.
2021-09-10 18:01:00,999 [canton-env-execution-context-18] INFO  akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2021-09-10 18:01:01,047 [main] INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
2021-09-10 18:01:01,143 [main] DEBUG io.grpc.NameResolverRegistry - Service loader found io.grpc.internal.DnsNameResolverProvider@21f459fc
2021-09-10 18:01:01,174 [main] INFO  c.d.c.e.EnterpriseEnvironment tid:e12838ac803568950bf919dfd819276c - Manual start requested.

No difference, unfortunately.

DA
davidpadbury
Sep 2021

What’s a bit odd is were not getting as far as we were earlier in this thread.

Let’s try:

./bin/canton daemon -c examples/01-simple-topology/simple-topology.conf --bootstrap=examples/01-simple-topology/simple-ping.canton --log-file-name=$PWD/tmp/canton.log --debug
MA
Matheus
Sep 2021

Yeah, I also think it’s strange… before with a “wrong” java image I could see the interactive console but couldn’t start the domain, when I switched to openJDK 11 this started being the behavior.

with this

./bin/canton daemon -c examples/01-simple-topology/simple-topology.conf --bootstrap=examples/01-simple-topology/simple-ping.canton --log-file-name=$PWD/tmp/canton.log --debug

command now:

sh-4.4$ ./bin/canton daemon -c examples/01-simple-topology/simple-topology.conf --bootstrap=examples/01-simple-topology/simple-ping.canton --log-file-name=$PWD/tmp/canton.log --debug
INFO  c.d.c.CantonEnterpriseApp$ - Starting Canton version 0.27.0
INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
INFO  c.d.c.e.EnterpriseEnvironment - Creating ForkJoinPool with parallelism = 2 (instead of 1) to avoid starvation.
INFO  a.e.s.Slf4jLogger - Slf4jLogger started
INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
INFO  c.d.c.e.EnterpriseEnvironment tid:c7047fb7ee307f9886a513fae9ad95e4 - Manual start requested.
ERROR c.d.c.ServerRunner - An internal error occurred while executing script.
java.lang.ExceptionInInitializerError: null
        at ammonite.Main$.apply$default$5(Main.scala:68)
        at com.digitalasset.canton.console.HeadlessConsole$.createDefaultAmmoniteOptions(HeadlessConsole.scala:59)
        at com.digitalasset.canton.console.HeadlessConsole$.$anonfun$apply$1(HeadlessConsole.scala:46)
        at better.files.Dispose.apply(Dispose.scala:81)
        at com.digitalasset.canton.console.HeadlessConsole$.apply(HeadlessConsole.scala:45)
        at com.digitalasset.canton.ConsoleScriptRunner$.run(Runner.scala:169)
        at com.digitalasset.canton.ServerRunner.startWithBootstrap$1(Runner.scala:51)
        at com.digitalasset.canton.ServerRunner.$anonfun$run$3(Runner.scala:53)
        at com.digitalasset.canton.ServerRunner.$anonfun$run$3$adapted(Runner.scala:53)
        at scala.Option.fold(Option.scala:263)
        at com.digitalasset.canton.ServerRunner.run(Runner.scala:53)
        at com.digitalasset.canton.CantonAppDriver.delayedEndpoint$com$digitalasset$canton$CantonAppDriver$1(CantonAppDriver.scala:145)
        at com.digitalasset.canton.CantonAppDriver$delayedInit$body.apply(CantonAppDriver.scala:28)
        at scala.Function0.apply$mcV$sp(Function0.scala:39)
        at scala.Function0.apply$mcV$sp$(Function0.scala:39)
        at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
        at scala.App.$anonfun$main$1(App.scala:76)
        at scala.App.$anonfun$main$1$adapted(App.scala:76)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:919)
        at scala.App.main(App.scala:76)
        at scala.App.main$(App.scala:74)
        at com.digitalasset.canton.CantonAppDriver.main(CantonAppDriver.scala:28)
        at com.digitalasset.canton.CantonEnterpriseApp.main(CantonEnterpriseApp.scala)
Caused by: java.lang.IllegalArgumentException: requirement failed: ? is not an absolute path
        at scala.Predef$.require(Predef.scala:338)
        at os.Path.<init>(Path.scala:430)
        at os.Path$.apply(Path.scala:388)
        at os.package$.<clinit>(package.scala:19)
        ... 25 common frames omitted
ERROR c.d.c.ServerRunner - Unexpected error while running server: null
INFO  c.d.c.ServerRunner - Exception causing error is:
java.lang.ExceptionInInitializerError: null
        at ammonite.Main$.apply$default$5(Main.scala:68)
        at com.digitalasset.canton.console.HeadlessConsole$.createDefaultAmmoniteOptions(HeadlessConsole.scala:59)
        at com.digitalasset.canton.console.HeadlessConsole$.$anonfun$apply$1(HeadlessConsole.scala:46)
        at better.files.Dispose.apply(Dispose.scala:81)
        at com.digitalasset.canton.console.HeadlessConsole$.apply(HeadlessConsole.scala:45)
        at com.digitalasset.canton.ConsoleScriptRunner$.run(Runner.scala:169)
        at com.digitalasset.canton.ServerRunner.startWithBootstrap$1(Runner.scala:51)
        at com.digitalasset.canton.ServerRunner.$anonfun$run$3(Runner.scala:53)
        at com.digitalasset.canton.ServerRunner.$anonfun$run$3$adapted(Runner.scala:53)
        at scala.Option.fold(Option.scala:263)
        at com.digitalasset.canton.ServerRunner.run(Runner.scala:53)
        at com.digitalasset.canton.CantonAppDriver.delayedEndpoint$com$digitalasset$canton$CantonAppDriver$1(CantonAppDriver.scala:145)
        at com.digitalasset.canton.CantonAppDriver$delayedInit$body.apply(CantonAppDriver.scala:28)
        at scala.Function0.apply$mcV$sp(Function0.scala:39)
        at scala.Function0.apply$mcV$sp$(Function0.scala:39)
        at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
        at scala.App.$anonfun$main$1(App.scala:76)
        at scala.App.$anonfun$main$1$adapted(App.scala:76)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:919)
        at scala.App.main(App.scala:76)
        at scala.App.main$(App.scala:74)
        at com.digitalasset.canton.CantonAppDriver.main(CantonAppDriver.scala:28)
        at com.digitalasset.canton.CantonEnterpriseApp.main(CantonEnterpriseApp.scala)
Caused by: java.lang.IllegalArgumentException: requirement failed: ? is not an absolute path
        at scala.Predef$.require(Predef.scala:338)
        at os.Path.<init>(Path.scala:430)
        at os.Path$.apply(Path.scala:388)
        at os.package$.<clinit>(package.scala:19)
        ... 25 common frames omitted
DA
davidpadbury
Sep 2021

Okay, progress! Well back to the original problem…

So that still looks like the new temporary path isnt being picked up. Could you do the following just to really make sure it’s picking up our new temporary file path:

./bin/canton daemon -Djava.io.tmpdir=/home/app/temp -c examples/01-simple-topology/simple-topology.conf --bootstrap=examples/01-simple-topology/simple-ping.canton --log-file-name=$PWD/tmp/canton.log --debug
MA
Matheus
Sep 2021

Okay, so this is different:
when running the exact command you sent:

./bin/canton daemon -Djava.io.tmpdir=/home/app/temp -c examples/01-simple-topology/simple-topology.conf --bootstrap=examples/01-simple-topology/simple-ping.canton --log-file-name=$PWD/tmp/canton.log --debug

we get:

INFO  c.d.c.CantonEnterpriseApp$ - Starting Canton version 0.27.0
INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
INFO  c.d.c.e.EnterpriseEnvironment - Creating ForkJoinPool with parallelism = 2 (instead of 1) to avoid starvation.
INFO  a.e.s.Slf4jLogger - Slf4jLogger started
INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
INFO  c.d.c.e.EnterpriseEnvironment tid:3d72cae7793ab98823e8071e6d7fdbcc - Manual start requested.
ERROR c.d.c.ServerRunner - An internal error occurred while executing script.
java.nio.file.NoSuchFileException: /home/app/temp/ammonite_cache8685047869954133097
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
        at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389)        at java.base/java.nio.file.Files.createDirectory(Files.java:690)
        at java.base/java.nio.file.TempFileHelper.create(TempFileHelper.java:135)
        at java.base/java.nio.file.TempFileHelper.createTempDirectory(TempFileHelper.java:172)
        at java.base/java.nio.file.Files.createTempDirectory(Files.java:1007)
        at better.files.File$.newTemporaryDirectory(File.scala:1360)
        at better.files.File$.temporaryDirectory(File.scala:1369)
        at com.digitalasset.canton.console.HeadlessConsole$.apply(HeadlessConsole.scala:45)
        at com.digitalasset.canton.ConsoleScriptRunner$.run(Runner.scala:169)
        at com.digitalasset.canton.ServerRunner.startWithBootstrap$1(Runner.scala:51)
        at com.digitalasset.canton.ServerRunner.$anonfun$run$3(Runner.scala:53)
        at com.digitalasset.canton.ServerRunner.$anonfun$run$3$adapted(Runner.scala:53)
        at scala.Option.fold(Option.scala:263)
        at com.digitalasset.canton.ServerRunner.run(Runner.scala:53)
        at com.digitalasset.canton.CantonAppDriver.delayedEndpoint$com$digitalasset$canton$CantonAppDriver$1(CantonAppDriver.scala:145)
        at com.digitalasset.canton.CantonAppDriver$delayedInit$body.apply(CantonAppDriver.scala:28)
        at scala.Function0.apply$mcV$sp(Function0.scala:39)
        at scala.Function0.apply$mcV$sp$(Function0.scala:39)
        at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
        at scala.App.$anonfun$main$1(App.scala:76)
        at scala.App.$anonfun$main$1$adapted(App.scala:76)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:919)
        at scala.App.main(App.scala:76)
        at scala.App.main$(App.scala:74)
        at com.digitalasset.canton.CantonAppDriver.main(CantonAppDriver.scala:28)
        at com.digitalasset.canton.CantonEnterpriseApp.main(CantonEnterpriseApp.scala)
ERROR c.d.c.ServerRunner - Unexpected error while running server: /home/app/temp/ammonite_cache8685047869954133097
INFO  c.d.c.ServerRunner - Exception causing error is:
java.nio.file.NoSuchFileException: **/home/app/temp/ammonite_cache8685047869954133097**
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
        at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389)
        at java.base/java.nio.file.Files.createDirectory(Files.java:690)
        at java.base/java.nio.file.TempFileHelper.create(TempFileHelper.java:135)
        at java.base/java.nio.file.TempFileHelper.createTempDirectory(TempFileHelper.java:172)
        at java.base/java.nio.file.Files.createTempDirectory(Files.java:1007)
        at better.files.File$.newTemporaryDirectory(File.scala:1360)
        at better.files.File$.temporaryDirectory(File.scala:1369)
        at com.digitalasset.canton.console.HeadlessConsole$.apply(HeadlessConsole.scala:45)
        at com.digitalasset.canton.ConsoleScriptRunner$.run(Runner.scala:169)
        at com.digitalasset.canton.ServerRunner.startWithBootstrap$1(Runner.scala:51)
        at com.digitalasset.canton.ServerRunner.$anonfun$run$3(Runner.scala:53)
        at com.digitalasset.canton.ServerRunner.$anonfun$run$3$adapted(Runner.scala:53)
        at scala.Option.fold(Option.scala:263)
        at com.digitalasset.canton.ServerRunner.run(Runner.scala:53)
        at com.digitalasset.canton.CantonAppDriver.delayedEndpoint$com$digitalasset$canton$CantonAppDriver$1(CantonAppDriver.scala:145)
        at com.digitalasset.canton.CantonAppDriver$delayedInit$body.apply(CantonAppDriver.scala:28)
        at scala.Function0.apply$mcV$sp(Function0.scala:39)
        at scala.Function0.apply$mcV$sp$(Function0.scala:39)
        at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
        at scala.App.$anonfun$main$1(App.scala:76)
        at scala.App.$anonfun$main$1$adapted(App.scala:76)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:919)
        at scala.App.main(App.scala:76)
        at scala.App.main$(App.scala:74)
        at com.digitalasset.canton.CantonAppDriver.main(CantonAppDriver.scala:28)
        at com.digitalasset.canton.CantonEnterpriseApp.main(CantonEnterpriseApp.scala)

showing that the path got picked up. However the folder name is tmp, so I changed that in the following command and:

sh-4.4$ bin/canton daemon -Djava.io.tmpdir=/home/app/tmp -c examples/01-simple-topology/simple-topology.conf --bootstrap=examples/01-simple-topology/simple-ping.canton --log-file-name=$PWD/tmp/canton.log --debug
INFO  c.d.c.CantonEnterpriseApp$ - Starting Canton version 0.27.0
INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
INFO  c.d.c.e.EnterpriseEnvironment - Creating ForkJoinPool with parallelism = 2 (instead of 1) to avoid starvation.
INFO  a.e.s.Slf4jLogger - Slf4jLogger started
INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
INFO  c.d.c.e.EnterpriseEnvironment tid:de2f7115e072bddf7b897dcabed0fbbc - Manual start requested.
ERROR c.d.c.ServerRunner - An internal error occurred while executing script.
java.lang.ExceptionInInitializerError: null
        at ammonite.Main$.apply$default$5(Main.scala:68)
        at com.digitalasset.canton.console.HeadlessConsole$.createDefaultAmmoniteOptions(HeadlessConsole.scala:59)
        at com.digitalasset.canton.console.HeadlessConsole$.$anonfun$apply$1(HeadlessConsole.scala:46)
        at better.files.Dispose.apply(Dispose.scala:81)
        at com.digitalasset.canton.console.HeadlessConsole$.apply(HeadlessConsole.scala:45)
        at com.digitalasset.canton.ConsoleScriptRunner$.run(Runner.scala:169)
        at com.digitalasset.canton.ServerRunner.startWithBootstrap$1(Runner.scala:51)
        at com.digitalasset.canton.ServerRunner.$anonfun$run$3(Runner.scala:53)
        at com.digitalasset.canton.ServerRunner.$anonfun$run$3$adapted(Runner.scala:53)
        at scala.Option.fold(Option.scala:263)
        at com.digitalasset.canton.ServerRunner.run(Runner.scala:53)
        at com.digitalasset.canton.CantonAppDriver.delayedEndpoint$com$digitalasset$canton$CantonAppDriver$1(CantonAppDriver.scala:145)
        at com.digitalasset.canton.CantonAppDriver$delayedInit$body.apply(CantonAppDriver.scala:28)
        at scala.Function0.apply$mcV$sp(Function0.scala:39)
        at scala.Function0.apply$mcV$sp$(Function0.scala:39)
        at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
        at scala.App.$anonfun$main$1(App.scala:76)
        at scala.App.$anonfun$main$1$adapted(App.scala:76)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:919)
        at scala.App.main(App.scala:76)
        at scala.App.main$(App.scala:74)
        at com.digitalasset.canton.CantonAppDriver.main(CantonAppDriver.scala:28)
        at com.digitalasset.canton.CantonEnterpriseApp.main(CantonEnterpriseApp.scala)
Caused by: java.lang.IllegalArgumentException: requirement failed: ? is not an absolute path
        at scala.Predef$.require(Predef.scala:338)
        at os.Path.<init>(Path.scala:430)
        at os.Path$.apply(Path.scala:388)
        at os.package$.<clinit>(package.scala:19)
        ... 25 common frames omitted
ERROR c.d.c.ServerRunner - Unexpected error while running server: null
INFO  c.d.c.ServerRunner - Exception causing error is:
java.lang.ExceptionInInitializerError: null
        at ammonite.Main$.apply$default$5(Main.scala:68)
        at com.digitalasset.canton.console.HeadlessConsole$.createDefaultAmmoniteOptions(HeadlessConsole.scala:59)
        at com.digitalasset.canton.console.HeadlessConsole$.$anonfun$apply$1(HeadlessConsole.scala:46)
        at better.files.Dispose.apply(Dispose.scala:81)
        at com.digitalasset.canton.console.HeadlessConsole$.apply(HeadlessConsole.scala:45)
        at com.digitalasset.canton.ConsoleScriptRunner$.run(Runner.scala:169)
        at com.digitalasset.canton.ServerRunner.startWithBootstrap$1(Runner.scala:51)
        at com.digitalasset.canton.ServerRunner.$anonfun$run$3(Runner.scala:53)
        at com.digitalasset.canton.ServerRunner.$anonfun$run$3$adapted(Runner.scala:53)
        at scala.Option.fold(Option.scala:263)
        at com.digitalasset.canton.ServerRunner.run(Runner.scala:53)
        at com.digitalasset.canton.CantonAppDriver.delayedEndpoint$com$digitalasset$canton$CantonAppDriver$1(CantonAppDriver.scala:145)
        at com.digitalasset.canton.CantonAppDriver$delayedInit$body.apply(CantonAppDriver.scala:28)
        at scala.Function0.apply$mcV$sp(Function0.scala:39)
        at scala.Function0.apply$mcV$sp$(Function0.scala:39)
        at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
        at scala.App.$anonfun$main$1(App.scala:76)
        at scala.App.$anonfun$main$1$adapted(App.scala:76)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:919)
        at scala.App.main(App.scala:76)
        at scala.App.main$(App.scala:74)
        at com.digitalasset.canton.CantonAppDriver.main(CantonAppDriver.scala:28)
        at com.digitalasset.canton.CantonEnterpriseApp.main(CantonEnterpriseApp.scala)
Caused by: java.lang.IllegalArgumentException: requirement failed: ? is not an absolute path
        at scala.Predef$.require(Predef.scala:338)
        at os.Path.<init>(Path.scala:430)
        at os.Path$.apply(Path.scala:388)
        at os.package$.<clinit>(package.scala:19)
        ... 25 common frames omitted

With this different exception it looks like if fails somewhere after writing to the folder.

DA
davidpadbury
Sep 2021

Sorry about suggesting .../temp instead of /tmp. My bad.

Okay, this library is trying to convert the current working directory (pwd) to an absolute path and somehow not producing an absolute path. Could you show me echo $PWD just out of interest?

This is coming from the portion trying to run the bootstrap script so lets instead try to load the console now we have the temp dir sorted out but without loading our fancy repl (the --no-tty flag):

bin/canton daemon --no-tty -Djava.io.tmpdir=/home/app/tmp -c examples/01-simple-topology/simple-topology.conf --log-file-name=$PWD/tmp/canton.log --debug
RA
Ratko_Veprek
Sep 2021

@Matheus

Please check the documentation on how to run scripts remotely. It’s quite easy:
https://www.canton.io/docs/dev/user-manual/usermanual/console.html#remote-administration

On the other note: Right now, there are three issues:
(1) ammonite caching between process restarts
(2) ammonite struggling to determine pwd
(3) ammonite requiring ability to create temp files

The “requirement failed: ? is not an absolute path” is related to (2). With the version you have, you can currently not work around that issue. Here, the following java code seems to produce an invalid result:

java.nio.file.Paths.get(".").toAbsolutePath

I’ve modified Canton (not released yet) to make (1) and (2) manually configurable and allow in-memory caching for (1) and manually configure what the pwd is for (2). I’ll point you to the “snapshot release” once I merged the code.

In order to fix (3) we need to make sure that

java.io.File.createTempFile()

works on your setup, which is what @davidpadbury tried to suggest.

Therefore, my suggestion now is:
(1) please test from your side that temp file creation works.
(2) we’ll let you know where the daily snapshot versions of the community edition are so that you can test out the fix.

Cheers,
Ratko

MA
Matheus
Sep 2021

Hi guys,

@davidpadbury so this looks more promising.

So firstly echo $PWD seems to work fine:

sh-4.4$ echo $PWD
/home/app

Secondly, running with --no-tty gives us much more feedback:

INFO  c.d.c.CantonEnterpriseApp$ - Starting Canton version 0.27.0
INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
INFO  c.d.c.e.EnterpriseEnvironment - Creating ForkJoinPool with parallelism = 2 (instead of 1) to avoid starvation.
INFO  a.e.s.Slf4jLogger - Slf4jLogger started
INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
INFO  c.d.c.e.EnterpriseEnvironment tid:de2ea15e0626895cd8bc5babf15ac3f6 - Automatically starting all instances
INFO  c.d.c.p.EnterpriseParticipantNodeFactory$$anon$1:participant=participant1 - Starting admin-api services on AdminServerConfig(127.0.0.1,6012,None,Some(KeepAliveConfig(40s,20s)),10485760)
INFO  c.d.c.p.EnterpriseParticipantNodeFactory$$anon$1:participant=participant1 - Node is not initialized yet. Performing automated default initialization.
INFO  c.d.c.p.EnterpriseParticipantNodeFactory$$anon$1:participant=participant1 - Initializing node with id NodeId(participant1::1220e9bba026c7e607ac2cddb652675a3b61c16fff743eb64b01de8cc9ccb9efa09d,7fc7fbf8-be14-4c03-8e82-84c9548238c5)
INFO  c.d.c.p.i.ParticipantIdentityManager:participant=participant1 tid:2702a61d824f122cae276fca452fb6fb - Applied identity transaction NamespaceDelegation(
  namespace = 1220e9bba026c7e607ac2cddb652675a3b61c16fff743eb64b01de8cc9ccb9efa09d,
  target = SigningPublicKey(id = 1220e9bba026c7e607ac2cddb652675a3b61c16fff743eb64b01de8cc9ccb9efa09d, format = Tink, scheme = Ed25519),
  isRootDelegation = true
) at 2021-09-11T12:36:19.063309Z
INFO  c.d.c.p.p.PruningProcessorImpl:participant=participant1 - Pruning status: ParticipantPruningStatus()
INFO  c.d.c.p.l.a.CantonLedgerApiServerWrapper$Config:participant=participant1 - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
INFO  c.d.c.p.l.a.CantonLedgerApiServerWrapper$Config:participant=participant1 - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
INFO  c.d.p.s.d.HikariConnection - Attempting to connect to the database (attempt 1/600), context: {participant: "participant1"}
INFO  c.z.h.HikariDataSource - daml.index.db.connection.migrations - Starting...
INFO  c.z.h.HikariDataSource - daml.index.db.connection.migrations - Start completed.
INFO  c.d.p.s.FlywayMigrationsCanton - Ensuring Flyway migration has either not started or there are no pending migrations..., context: {participant: "participant1"}
INFO  o.f.c.i.l.VersionPrinter - Flyway Community Edition 7.8.1 by Redgate
INFO  o.f.c.i.d.b.DatabaseType - Database: jdbc:h2:mem:ledger_api_participant11220e9bba026c7e607ac2cddb652675a3b61c16fff743eb64b01de8cc9ccb9efa09d_a4604687697c4edab06b0b8e07e58ec7 (H2 1.4)
INFO  c.d.p.s.FlywayMigrationsCanton - Running Flyway migration on empty database with 33 migrations pending..., context: {participant: "participant1"}
INFO  o.f.c.i.l.VersionPrinter - Flyway Community Edition 7.8.1 by Redgate
INFO  o.f.c.i.c.DbValidate - Successfully validated 33 migrations (execution time 00:00.014s)
INFO  o.f.c.i.s.JdbcTableSchemaHistory - Creating Schema History table "PUBLIC"."flyway_schema_history" ...
INFO  o.f.c.i.c.DbMigrate - Current version of schema "PUBLIC": << Empty Schema >>
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "1 - Init"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "2 - Command deduplication"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "3 - Contract Divulgence"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "4 - Contract Divulgence"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "5.0 - Extract Event Data"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "5.1 - Populate Event Data"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "5.2 - Extract Event Data"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "6 - Disclosures index"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "7 - Add configuration"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "8 - Party entries"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "9 - Package entries"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "10 - Loosen transaction check"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "11 - Create command completions table"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "12 - Command deduplication"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "13 - Events new schema"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "14 - Stable offsets"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "15 - Remove maximum record time"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "16 - Delete checkpoints"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "17 - Stable offsets archival"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "18 - Contracts new schema"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "19 - Events table fixes"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "20 - Drop old schema"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "21 - Event witnesses single table"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "22 - Add witnesses to participant events"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "23 - Parties is local"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "24 - event sequential id"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "25 - drop participant id"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "26 - add participant id to parameters"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "27 - Participant-pruning"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "28 - multiple submitters"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "29 - explicit compression"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "30 - offset as text"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "99 - Contract keys canton"
INFO  o.f.c.i.c.DbMigrate - Successfully applied 33 migrations to schema "PUBLIC", now at version v99 (execution time 00:00.620s)
INFO  c.d.p.s.FlywayMigrationsCanton - Flyway schema migration finished successfully, applying org.flywaydb.core.api.output.MigrateResult@7cf6cd1 steps on empty database., context: {participant: "participant1"}
INFO  c.z.h.HikariDataSource - daml.index.db.connection.migrations - Shutdown initiated...
INFO  c.z.h.HikariDataSource - daml.index.db.connection.migrations - Shutdown completed.
INFO  c.d.p.i.RecoveringIndexer - Starting Indexer Server, context: {participant: "participant1"}
INFO  c.d.p.s.d.HikariConnection - Attempting to connect to the database (attempt 1/600), context: {participant: "participant1"}
INFO  c.z.h.HikariDataSource - daml.index.db.connection.indexer - Starting...
INFO  c.z.h.HikariDataSource - daml.index.db.connection.indexer - Start completed.
INFO  c.d.p.i.JdbcIndexerCanton$Factory - Initializing ledger with ID: participant1, context: {participant: "participant1"}
INFO  c.d.p.i.RecoveringIndexer - Started Indexer Server, context: {participant: "participant1"}
INFO  c.d.p.s.d.JdbcLedgerDaoCanton - Storing configuration entry, context: {participant: "participant1", submissionId: "TimeModel config"}
INFO  c.d.p.s.d.HikariConnection - Attempting to connect to the database (attempt 1/600), context: {participant: "participant1"}
INFO  c.z.h.HikariDataSource - daml.index.db.connection.api-server - Starting...
INFO  c.z.h.HikariDataSource - daml.index.db.connection.api-server - Start completed.
INFO  c.d.p.i.ReadOnlySqlLedger - Found existing ledger with ID: participant1, context: {participant: "participant1"}
INFO  c.d.p.a.ApiServicesCanton - Daml-LF Engine supports LF versions: 1.14, context: {participant: "participant1"}
INFO  c.d.p.a.s.LedgerConfigProvider - Initial ledger configuration lookup found configuration Configuration(1,LedgerTimeModel(PT0S,PT8760H,PT8760H),PT24H) at Absolute(000000000000000001). Looking for new ledger configurations from this offset., context: {participant: "participant1"}
INFO  c.d.p.a.LedgerApiServer - Listening on 127.0.0.1:6011 over plain text., context: {participant: "participant1"}
INFO  c.d.p.a.StandaloneApiServerCanton - Initialized API server version 1.16.0 with ledger-id = participant1, port = 6011, dar file = List(), context: {participant: "participant1"}
INFO  c.d.p.s.d.JdbcLedgerDaoCanton - Storing package entry, context: {participant: "participant1"}
INFO  c.d.c.p.i.ParticipantIdentityManager:participant=participant1 tid:2f713e9acba764f0ff37bd8df95bce76 - Applied identity transaction VettedPackages(
  participant = participant1::1220e9bba026c7e607ac2cddb652675a3b61c16fff743eb64b01de8cc9ccb9efa09d,
  packages = Seq(
    65921e553a35...,
    57b5c520512c...,
    cb0552debf21...,
    3f4deaf145a1...,
    86828b984346...,
    f20de1e4e37b...,
    76bf0fd12bd9...,
    d58cf9939847...,
    40f452260bef...,
    e491352788e5...,
    6839a6d3d430...,
    518032f41fd0...,
    852d8e3a8ccf...,
    bfcd37bd6b84...,
    cc348d369011...,
    057eed1fd48c...,
    d14e08374fc7...,
    c1f1f0055879...,
    6c2c0667393c...,
    e22bce619ae2...,
    8a7806365bbd...,
    97b883cd8a2b...,
    733e38d36a27...,
    99a2705ed38c...
  )
) at 2021-09-11T12:36:23.340667Z
INFO  c.d.p.a.s.ApiLedgerIdentityService - Received request for ledger identity: GetLedgerIdentityRequest(), context: {participant: "participant1"}
INFO  c.d.p.a.s.ApiCommandCompletionService - Received request for completion subscription 0: CompletionStreamRequest(participant1,admin-ping,Set(participant1::1220e9bba026c7e607ac2cddb652675a3b61c16fff743eb64b01de8cc9ccb9efa09d),Some(Absolute(000000000000000002))), context: {participant: "participant1", parties: ["participant1::1220e9bba026c7e607ac2cddb652675a3b61c16fff743eb64b01de8cc9ccb9efa09d"], offset: "000000000000000002"}
INFO  c.d.p.a.s.ApiLedgerIdentityService - Received request for ledger identity: GetLedgerIdentityRequest(), context: {participant: "participant1"}
INFO  c.d.p.a.s.t.ApiTransactionService - Received request for transactions., context: {participant: "participant1", startExclusive: "000000000000000002", endInclusive: null, filters: {participant1::1220e9bba026c7e607ac2cddb652675a3b61c16fff743eb64b01de8cc9ccb9efa09d: "all-templates"}, verbose: false, ledgerId: "participant1"}
INFO  c.d.p.a.s.ApiCommandCompletionService - Received request for completion subscription 1: CompletionStreamRequest(participant1,admin-dar-distribution,Set(participant1::1220e9bba026c7e607ac2cddb652675a3b61c16fff743eb64b01de8cc9ccb9efa09d),Some(Absolute(000000000000000002))), context: {participant: "participant1", parties: ["participant1::1220e9bba026c7e607ac2cddb652675a3b61c16fff743eb64b01de8cc9ccb9efa09d"], offset: "000000000000000002"}
INFO  c.d.p.a.s.t.ApiTransactionService - Received request for transactions., context: {participant: "participant1", startExclusive: "000000000000000002", endInclusive: null, filters: {participant1::1220e9bba026c7e607ac2cddb652675a3b61c16fff743eb64b01de8cc9ccb9efa09d: "all-templates"}, verbose: false, ledgerId: "participant1"}
INFO  c.d.c.p.i.ParticipantIdentityManager:participant=participant1 tid:2702a61d824f122cae276fca452fb6fb - Applied identity transaction OwnerToKeyMapping(
  owner = PAR::participant1::1220e9bba026c7e607ac2cddb652675a3b61c16fff743eb64b01de8cc9ccb9efa09d,
  key = SigningPublicKey(id = 122043951d8c04df595c2531b9653cef30544d30fc8fe508abf8889f84270ff01585, format = Tink, scheme = Ed25519)
) at 2021-09-11T12:36:23.915756Z
INFO  c.d.c.p.i.ParticipantIdentityManager:participant=participant1 tid:2702a61d824f122cae276fca452fb6fb - Applied identity transaction OwnerToKeyMapping(
  owner = PAR::participant1::1220e9bba026c7e607ac2cddb652675a3b61c16fff743eb64b01de8cc9ccb9efa09d,
  key = EncryptionPublicKey(id = 122081ce8ee7ba7384975e718f89189b141738998a00b9803fcd6ebda45f079cb4ac, format = Tink, scheme = ECIES-P256_HMAC256_AES128-GCM)
) at 2021-09-11T12:36:23.922569Z
INFO  c.d.c.p.i.ParticipantIdentityManager:participant=participant1 tid:2702a61d824f122cae276fca452fb6fb - Applied identity transaction SignedLegalIdentityClaim(signature = Signature(signature = 306602310084..., signedBy = 12208d7e6a9a31f2e089dc029cf9bf7a58a81ad6aad0d9332a9e085e09c4c592c659), claim = ...) at 2021-09-11T12:36:24.196814Z
INFO  c.d.c.p.s.CantonSyncService:participant=participant1 tid:de2ea15e0626895cd8bc5babf15ac3f6 - Reconnecting to domains List()
INFO  c.d.c.p.s.CantonSyncService:participant=participant1 tid:de2ea15e0626895cd8bc5babf15ac3f6 - Successfully re-connected to domains
INFO  c.d.c.e.EnterpriseEnvironment tid:de2ea15e0626895cd8bc5babf15ac3f6 - Successfully started all nodes
INFO  c.d.c.ServerRunner - Canton started

@Ratko_Veprek thanks for the link, I’ll take a look in the docs.

I’ll also do a test with java.io.File.createTempFile() using a separate java project and post the result here.

Thanks both for the great support!

Cheers,
Matheus

MA
Matheus
Sep 2021

Hello gentlemen,

So, although running a remote console locally was no problem, it has proven to be quite the struggle on our current setup due to the restrictions of our test environment.

Is there a ETA on the fix for the bootstrap script? I believe running that would take us most of the way we need to go.

Thanks in advance!

Best,
Matheus

RA
Ratko_Veprek
Sep 2021

Hi Matheus!

Here is a daily snapshot build that has an improvement with the ammonite caches:
https://www.canton.io/releases/canton-community-20210923.tar.gz

You have two additional configuration options:

canton.parameters.console.cache-dir = 
canton.parameters.console.working-dir =

By default, these parameters are inferred. But if you define them we should at least get around the “?” path issue.

Can you then run

bin/canton daemon --no-tty -Djava.io.tmpdir=/home/app/tmp -c examples/01-simple-topology/simple-topology.conf --log-file-name=$PWD/tmp/canton.log --debug --bootstrap examples/01-simple-topology/simple-ping.canton

One of the last log lines should contain

INFO  c.d.c.p.p.TransactionProcessor:participant=participant2/domain=mydomain tid:.. - Finalizing Transaction request at RequestId(2021-09-24T10:00:14.931475Z) with event Some(TransactionAccepted(

If you see that then script execution works.

If that worked, then run the same command above again, but without daemon. Let’s hope you’ll get a console.

Best,
Ratko

MA
Matheus
Sep 2021

Hi Ratko,

Thanks for the response. So this is what I get now trying the 0.28.0 snapshot:

sh-4.4$ bin/canton daemon --no-tty -Djava.io.tmpdir=/home/app/tmp -c newconf.conf --log-file-name=$PWD/tmp/canton.log --debug
INFO  c.d.c.CantonCommunityApp$ - Starting Canton version 0.28.0-SNAPSHOT
Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: ? is not an absolute path
        at scala.Predef$.require(Predef.scala:338)
        at os.Path.<init>(Path.scala:430)
        at os.Path$.apply(Path.scala:388)
        at ammonite.main.Defaults$.ammoniteHome(Defaults.scala:32)
        at com.digitalasset.canton.console.AmmoniteConsoleConfig$.defaultCacheDir(AmmoniteConsoleConfig.scala:30)
        at com.digitalasset.canton.console.AmmoniteConsoleConfig$.apply$default$1(AmmoniteConsoleConfig.scala:22)
        at com.digitalasset.canton.config.CantonParameters$.apply$default$8(CantonConfig.scala:256)
        at com.digitalasset.canton.config.CantonCommunityConfig$.apply$default$6(CantonCommunityConfig.scala:31)
        at com.digitalasset.canton.config.CantonCommunityConfig$anon$lazy$macro$194$1.inst$macro$138$lzycompute(CantonCommunityConfig.scala:67)
        at com.digitalasset.canton.config.CantonCommunityConfig$anon$lazy$macro$194$1.inst$macro$138(CantonCommunityConfig.scala:67)
        at com.digitalasset.canton.config.CantonCommunityConfig$.cantonCommunityConfigReader$lzycompute(CantonCommunityConfig.scala:67)
        at com.digitalasset.canton.config.CantonCommunityConfig$.cantonCommunityConfigReader(CantonCommunityConfig.scala:61)
        at com.digitalasset.canton.config.CantonCommunityConfig$.load(CantonCommunityConfig.scala:82)
        at com.digitalasset.canton.CantonCommunityApp$.loadConfig(CantonCommunityApp.scala:17)
        at com.digitalasset.canton.CantonAppDriver.delayedEndpoint$com$digitalasset$canton$CantonAppDriver$1(CantonAppDriver.scala:95)
        at com.digitalasset.canton.CantonAppDriver$delayedInit$body.apply(CantonAppDriver.scala:29)
        at scala.Function0.apply$mcV$sp(Function0.scala:39)
        at scala.Function0.apply$mcV$sp$(Function0.scala:39)
        at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
        at scala.App.$anonfun$main$1(App.scala:76)
        at scala.App.$anonfun$main$1$adapted(App.scala:76)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:919)
        at scala.App.main(App.scala:76)
        at scala.App.main$(App.scala:74)
        at com.digitalasset.canton.CantonAppDriver.main(CantonAppDriver.scala:29)
        at com.digitalasset.canton.CantonCommunityApp.main(CantonCommunityApp.scala)

I tried both variants below in the config file:

canton {
  parameters {
    manual-start = yes
    console.cache-dir = /home/app/tmp
    console.working-dir = /home/app/tmp
  }
  ....

canton {
  parameters {
    manual-start = yes
    console.cache-dir = "/home/app/tmp"
    console.working-dir = "/home/app/tmp"
  }
  ....

But the result is the same, maybe there’s another path it’s failing to determine?

Cheers,
Matheus

RA
Ratko_Veprek
Sep 2021

Can you try to additionally start with

./bin/canton -Duser.home=/home/app ...

It seems like

System.getProperty("user.home")

yields

?
MA
Matheus
Sep 2021

That did the trick :slight_smile: seems to be up and running, removing daemon made the interactive console appear.

Thanks a lot for the great support!

Can we expect these changes to be in the next Canton Enterprise release?

RA
Ratko_Veprek
Sep 2021

Yes. Enterprise is a highly enhanced version of the community release to with parallel transaction processing (much faster), high availability, ledger pruning. So generally, enterprise supports everything that community supports, but not the other way around.

← Back to Discussions