Does the karamel only support GitHub?

Hopsworks
#1

Because of some “cookbook-dependency” problems (Please refer to: Some "cookbook-dependency" problems of Hopsworks community edition 2.2 - #7 by Freeman), I suspect that my servers cannot access GitHub correctly. So I am planning to download the installation resources to the local server and continue installation based on hopsworks-installer.sh (2.2).

By reading the documents of Karamel, it seems that Karamel only supports GitHub. I am wondering if the Karamel could support GitLab?
Any comments will be much appreciated.

#2

I pulled and read the source code of Karamel, the Git branch is “master”. I found the following code in “se.kth.karamel.common.util.Settings” class:

public static final String GITHUB_BASE_URL_PATTERN = “http(?:s)?:\/\/github.com”;

It shows that Karamel only supports GitHub at present. It seems that the R&D team of Hopsworks have modified Karamel, since I didn’t find the 0.6 branch.

Another problem is that Karamel cannot create “.karamel” directory on a bare machine, I am not sure if this issue has anything to do with the JDK version. Because I have tried the Oracle jdk1.8.0_231 and the OpenJDK 1.8.0_282 respectively, neither of them solved this problem. I’ve also set the “SELinux” to “disabled”, but this issue still remains.

I’m looking forward to a reply, any comments will be much appreciated.

Some "cookbook-dependency" problems of Hopsworks community edition 2.2
#3

I added a line of code to output more log information in class “se.kth.karamel.common.cookbookmeta.Berksfile”, method is “validateGithubUrls()”. The main logic is as follows:

for (Map.Entry<String, Cookbook> entry : deps.entrySet()) {
  String name = entry.getKey();
  Cookbook cb = entry.getValue();
  String homeUrl = cb.getUrls().cookbookUrl;
  logger.info("homeUrl ---> " + homeUrl);
  String errorMsg = String.format("Cookbook-dependency '%s' doesn't refer to a valid url in Berksfile", name);
  try {
    if (validUrls.contains(homeUrl)) {
      logger.debug(String.format("Skip validating url '%s' since it was already validated", homeUrl));
    } else {
      logger.debug(String.format("Validating url '%s'", homeUrl));
      URL u = new URL(homeUrl);
      HttpURLConnection huc = (HttpURLConnection) u.openConnection();
      huc.setRequestMethod("GET");
      huc.connect();
      int code = huc.getResponseCode();
      if (code != 200) {
        throw new CookbookUrlException(errorMsg);
      }
      validUrls.add(homeUrl);
    }
  } catch (IOException ex) {
    throw new CookbookUrlException(errorMsg, ex);
  }
}

The content of the log is as follows:

INFO [2021-04-25 05:35:19,607] org.eclipse.jetty.server.handler.ContextHandler: Started i.d.j.MutableServletContextHandler@2899a8db{/,null,AVAILABLE}
INFO [2021-04-25 05:35:19,609] io.dropwizard.setup.AdminEnvironment: tasks =

POST    /tasks/gc (io.dropwizard.servlets.tasks.GarbageCollectionTask)

INFO [2021-04-25 05:35:19,614] org.eclipse.jetty.server.handler.ContextHandler: Started i.d.j.MutableServletContextHandler@62df0ff3{/admin,null,AVAILABLE}
INFO [2021-04-25 05:35:19,634] org.eclipse.jetty.server.ServerConnector: Started karamel-core@5b057c8c{HTTP/1.1}{0.0.0.0:9090}
INFO [2021-04-25 05:35:21,801] se.kth.karamel.client.api.CookbookCacheIml: 0-level cookbooks for Hops is 1
INFO [2021-04-25 05:35:22,983] se.kth.karamel.common.cookbookmeta.Berksfile: homeUrl —> GitHub - logicalclocks/airflow-chef: Airflow workflow management platform chef cookbook.
INFO [2021-04-25 05:35:24,222] se.kth.karamel.common.cookbookmeta.Berksfile: homeUrl —> GitHub - logicalclocks/kube-hops-chef
ERROR [2021-04-25 05:37:31,559] se.kth.karamel.webservice.KaramelServiceApplication: Inalid yaml file; Cookbook-dependency ‘kube-hops’ doesn’t refer to a valid url in Berksfile
INFO [2021-04-25 05:37:31,591] org.eclipse.jetty.server.ServerConnector: Stopped karamel-core@5b057c8c{HTTP/1.1}{0.0.0.0:9090}
INFO [2021-04-25 05:37:31,594] org.eclipse.jetty.server.handler.ContextHandler: Stopped i.d.j.MutableServletContextHandler@62df0ff3{/admin,null,UNAVAILABLE}
INFO [2021-04-25 05:37:31,595] org.eclipse.jetty.server.handler.ContextHandler: Stopped i.d.j.MutableServletContextHandler@2899a8db{/,null,UNAVAILABLE}

Obviously, the root cause of this issue is that some URL of Chef cannot be verified. But I can get a response by executing the command : curl GitHub - logicalclocks/kube-hops-chef.

It’s so weird … …

#4

About the latest reply, please refer to: Where can I get the source code of Karamel-0.6?

#5

Yes, it only supports Github at the moment. Closing this thread, if you have other questions about Karamel let’s use the other thread.

#6