Need documentation on how to access HopFS for all python (if possible), pyspark and spark

thaithien · March 27, 2020, 3:59am

I look into image of the tutorial and found a code for pyspark (in this tutorial .)

However, I don’t know how to use in spark notebook. For example, my concern is how to import.

import ? ;

tried:

I really need documentation, with some code example (copy paste that run, include import statement)

Steffen · March 27, 2020, 12:40pm

I cannot reproduce the error you are getting. Are you running the Jupyter notebook inside Hopsworks?

We have many example notebooks that can be found here: hops-examples

You can also run the Deep Learning tour available on the Hopsworks start page which will provide multiple example notebooks in Jupyter.

thaithien · April 3, 2020, 4:58am

that is scala notebook, not pyspark

Steffen · April 3, 2020, 7:09am

In this case check out some of the Scala examples:

github.com

logicalclocks/hops-examples/blob/master/spark/src/main/scala/io/hops/examples/spark/kafka/StructuredStreamingKafka.scala

package io.hops.examples.spark.kafka

import io.hops.util.Hops
import org.apache.spark.sql.avro._
import org.apache.spark.sql._
import java.text.SimpleDateFormat
import java.util.Date
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._

import org.apache.log4j.{Level, LogManager}
import org.apache.spark.SparkConf


/**
  * Example that uses HopsUtil and Spark-Kafka structured streaming with Avro.
  * Runs a producer and a consumer, depending on the user-provided argument.
  *
  */
object StructuredStreamingKafka {

This file has been truncated. show original

github.com

logicalclocks/hops-examples/blob/master/notebooks/featurestore/FeaturestoreTourScala.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " # Feature Store Tour - Scala API\n",
    " \n",
    "This notebook contains a tour/reference for the feature store Scala API on hopsworks. We will go over best practices for using the API as well as common pitfalls.\n",
    " \n",
    "The notebook is designed to be used in combination with the Feature Store Tour on Hopsworks, it assumes that you have run the following feature engineering job: [job](https://github.com/logicalclocks/hops-examples/tree/master/featurestore) (**the job is added automatically when you start the feature store tour in Hopsworks. You can run the job by going to the 'Jobs' tab to the left in the Hopsworks project home page**). \n",
    "\n",
    "Which will produce the following model of feature groups in your project's feature store:\n",
    "\n",
    "![Feature Store Model](./images/model.png \"Feature Store Model\")\n",
    "\n",
    "In this notebook we will run queries over this feature store model. We will also create new feature groups and training datasets.\n",
    "\n",
    "We will go from (1) features to (2) training datasets to (3) A trained model"
   ]

This file has been truncated. show original

And the Scala version of our library for some more documentation: https://github.com/logicalclocks/hops-util