<div id="readability-page-1" class="page"><div role="main">

 <article>

 



 

 

 <section>
 Introducing JupyterFlow, a better way to scale your ML job on Kubernetes.

I read this article “<a href="https://determined.ai/blog/data-scientists-dont-care-about-kubernetes">Data Scientists Don’t Care About Kubernetes</a>” and I totally agree with it. Using Kubernetes for training ML models is a great idea but it is not quite easy to use it, especially for Data Scientist. For those who haven’t read the article, it can be summed up as follows(I recommend you to read the origial article though):

<ul>
 <li>Kubernetes is a great tool for running ML model on multiple training server efficiently. There are many attempts to utilze Kubernetes in ML field(for example, Kubeflow).</li>
 <li>However, it is not quite friendly to use for data scientist because of Kubernetes’s steep learning curve. Kubernetes is made for software engineer, not data scientist.</li>
 <li>To make data scientist focus only on building ML model thoroughly, it is important to have a abstract ML tool that data scientist can easily utilze Kubernetes without knowing of it.</li>
</ul>

I also experienced the same problem and thought about a better way to run ML model on Kubernetes. Finally I came up with some idea, which might not be the best solution though, I feel it would fix the problem.



As the article pointed out, the reason why Kubernetes based ML tools such as Kubeflow is hard to use is two fold.

<ul>
 <li>Data scientist needs to containerize their ML code.</li>
 <li>Data scientist needs to write k8s manifest file(YAML).</li>
</ul>

Let’s first take a look at containerization problem. Writing one’s ML model into source code is not hard for data scientist. The problem starts when they want to deploy one’s code on Kubernetes for training or serving. To deploy a program onto Kubernetes, containerization needs to be done first. This process is for software engineer not for data scientist. It might be unnatural for data scientist. Writing Kubernetes manifest file is also burdensome. Data scientist needs to deal with detailed k8s concepts. There are some solutions for this problem.

<h2 id="solutions">Solutions</h2>

<h4 id="1-requesting-engineers-support">1. Requesting engineer’s support.</h4>

Software engineers can support containerizing data scientist’s ML code for them. It is the simplest and fastest solution. However each developer has their own work, so this method can not be sustainable and it will drag down the whole model development cycle. Moreover unlike software development, model development process has more fine-grained incremental steps. For example, you write your code and run right away for validation. It could easily be exhausting if there is a strong dependency between two group everytime the code changes.

<h4 id="2-learning-how-to-use-k8s">2. Learning how to use k8s.</h4>

As the aforementioned article states, everyone can not be a unicorn. Also, according to single-responsibility principle, it might be cost-efficient to focus on each of what they are good at(Actually, this principle is for programming, not for person though).

<h4 id="3-providing-abstract-ml-tool">3. Providing abstract ML tool.</h4>

The Determined AI’s approach is to provide an abstract way to run on Kubernetes through ML tool. They introduced their own product.

<h2 id="my-proposal">My proposal</h2>

I agree with Determined AI’s suggestion and I have a similar approach but slightly different. What do you say if I can to this:

<h3 id="removing-containerization-process-from-data-scientist">Removing containerization process from Data Scientist</h3>

The biggest pain point for data scientist to use Kubernetes based ML tool is containerizing their ML code. Then what if I could remove this step?

<img src="https://coffeewhale.com/assets/images/jupyterflow/whatthehell.jpg" alt="what-the-hell">

You might be thinking, What on earth is he talking about? In <a href="https://kubernetes.io/">Kubernetes website</a> first page, it states Kubernetes as “Production-Grade Container Orchestration”. How could you remove containerization step in the middle of utilizing Kubernetes as a ML platform?

I think, “it is possible”. Yet on one condition.

<h3 id="developing-model-inside-the-container-in-the-first-place">Developing model inside the container in the first place</h3>

Let me explain in more detail. Current problem is that data scientis should build their container by themself. Then, what if we make a container first and provide that containerized environment to data scientist? If data scientist writes their code inside the container in the first place, it would be much easier to run this code on Kubernetes. But there are some problems.

<ol>
 <li>How can we provide a containerized environment easily to data scientist?</li>
 <li>How can we deliver the ML code written inside a container to Kubernetes?</li>
</ol>

<h3 id="jupyterhub-on-kubernetes">JupyterHub on Kubernetes</h3>

The solution for the first problem is <a href="https://jupyterhub.readthedocs.io/">JupyterHub</a>. JupyterHub is a platform that each user can launch their own jupyter notebook server respectively. There are many methods to setup JupyterHub but my approach only works on <a href="https://zero-to-jupyterhub.readthedocs.io/en/latest/#setup-jupyterhub">JupyterHub on Kubernetes</a>. From now on, I wil refer to JupyterHub as JupyterHub on Kubernetes.

The architecture of JupyterHub is following.

<img src="https://coffeewhale.com/assets/images/jupyterflow/jupyterhub-arch.png" alt="JupyterHub">

It might look complicated but you only need to focus on <code>Spawners</code> and <code>Pod</code>. Everytime users launch their notebook, <code>Spawners</code> spawns a new <code>Pod</code> on Kubernetes. Each <code>Pod</code> represents jupyter notebook for each user. Each user writes their code in their notebook. Each <code>Pod</code> is connected to a shared storage, such as NAS(Network Attached Storage) server so every written code gathers to one place.

<ul>
 <li><code>Pod</code> == Jupyter notebook server</li>
 <li>ML code location == Shared Storage(NAS server)</li>
</ul>

Data scientist can write their ML code on this containerized Jupyter server.

<h3 id="delivering-the-code">Delivering the code</h3>

Next, we need to solve the second problem. Even if we manage to write a code inside a container, we still need to deliver that code to Kubernetes to run a job. Do we have to re-build an image from Jupyter notebook server? Not at all. There is a way to deliver the ML code without re-building the container.

Using Shared Storage simply solves the problem. Because the code, data scientist wrote, gets stored in Shared Storage(NAS), you only need to connect that same Shared Storage volume when you run a job on Kubernetes.

Let’s take a look at the basics of running ML code. There are three main parts. “Execution environment, ML code, Model hyper-parameter”.

<div><pre><code>venv/bin/python train.py epoch=10 dropout=0.5
</code></pre></div>

<ul>
 <li>Execution Env: <code>virtualenv</code> python environment(<code>venv</code>)</li>
 <li>ML code: <code>train.py</code></li>
 <li>Model H.P.: <code>epoch=10 dropout=0.5</code></li>
</ul>

If we could send these information to Kubernetes, data scientist can use k8s without any hard works. Suprisingly, you can get all these information from JupyterHub. Using the metadata of jupyter notebook <code>Pod</code> is the key.

<ul>
 <li>Execution Env: jupyter notebook container image (<code>Pod.spec.containers.image</code>)</li>
 <li>ML code: ML code located on Shared volume (<code>Pod.spec.volumes</code>)</li>
 <li>Model H.P.: parameter passed to ML tool</li>
</ul>

<img src="https://coffeewhale.com/assets/images/jupyterflow/newpod.png" alt="JupyterHub">

What if there is a ML tool which is smart enough to find the container image, ML code and H.P. from jupyter notebook <code>Pod</code> and constructs a Kubernetes manifest file and send it to k8s master, then data scientist can run their ML code without containerization job &amp; writing k8s manifest.

Is there really such a tool that can do this?

<hr>



<h3 id="introducing-jupyterflow">Introducing JupyterFlow</h3>

Introducing <a href="https://jupyterflow.com/">JupyterFlow (https://jupyterflow.com)</a>, a better way to scale your ML job on Kubernetes.

<img src="https://coffeewhale.com/assets/images/jupyterflow/side.png" alt="jupyterflow">

JupyterFlow is an Machine Learning CLI tool installed in jupyter notebook which helps you run ML code on Kubernetes without containerization &amp; manifest.

For example, write your code(<code>hello.py</code>, <code>world.py</code>) as you wish, run your code with <code>jupyterflow</code> CLI and JupyterFlow will create a ML pipeline for you.

<div><pre><code># write code. hello.py &amp; wrold.py
echo "print('hello')" &gt; hello.py
echo "print('world')" &gt; world.py

# install jupyterflow.
pip install jupyterflow

# in jupyterflow `&gt;&gt;` directive expresses container dependencies similar to Airflow.
jupyterflow run -c "python hello.py &gt;&gt; python world.py"
</code></pre></div>

The result looks like below:

<img src="https://raw.githubusercontent.com/hongkunyoo/jupyterflow/main/docs/images/intro.png" alt="results">

Think of model development cycle without JupyterFlow:

<ul>
 <li>Data scientist writes their code.</li>
 <li>Containerize their code.</li>
 <li>Write k8s manifest file.</li>
 <li>Submit job through <code>kubectl</code> CLI.</li>
</ul>

<img src="https://coffeewhale.com/assets/images/jupyterflow/painful.jpg" alt="">

On the otherhand, model development cycle with JupyterFlow becomes piece of cake:

<ul>
 <li>Spawn jupyter notebook.</li>
 <li>Write ML code on notebook.</li>
 <li>Run your ML pipeline through <code>jupyterflow</code>.</li>
</ul>

<img src="https://coffeewhale.com/assets/images/jupyterflow/solved.jpg" alt="">

Then JupyterFlow will do the rest. How about it, isn’t it simple?

<h3 id="jupyterflow-architecture">JupyterFlow Architecture</h3>

This is the architecture of JupyterFlow.

<img src="https://coffeewhale.com/assets/images/jupyterflow/architecture.png" alt="jupyterflow Architecture">

To use JupyterFlow, you need two main components. JupyterHub and <a href="https://argoproj.github.io/argo-workflows">Argo Workflow</a>. Argo Workflow is a <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources">custom controller</a> which let user to define dependencies between containers to make a workflow. You can use a new CRD(CustomResourceDefinition) called <code>Workflow</code> when you install Argo Workflow. Simple <code>Workflow</code> example looks like this:

<div><pre><code>apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
 generateName: hello-world-
spec:
 entrypoint: whalesay
 templates:
 - name: whalesay
 container:
 image: docker/whalesay
 command: [cowsay]
 args: ["hello world"]
 resources:
 limits:
 memory: 32Mi
 cpu: 100m
</code></pre></div>

If you write this manifest, custom controller(argo workflow controller) reads it and creates containers sequentially based on the instruction. Users can check the result from Argo Web UI (argo-ui) provided by Argo Workflow.

JupyterFlow’s role is to fetch jupyter notebook’s meta data(image, volume) and H.P. from user, and write <code>Workflow</code> manifest similar to this example. Then Kubernetes will run the ML pipeline for you.

<h3 id="compare-with-zeppelin--spark">Compare with Zeppelin &amp; Spark</h3>

Let’s think about Zeppelin and Spark to help you understand more.

<img src="https://coffeewhale.com/assets/images/jupyterflow/zeppelin-spark.png" alt="zeppelin spark">

Data engineer can use <code>Zeppelin</code> for interactive programming and submit Spark job through <code>spark-submit</code> even if they do not know all the details of Spark &amp; Hadoop cluster.

<img src="https://coffeewhale.com/assets/images/jupyterflow/jupyterflow-k8s.png" alt="zeppelin spark">

Likewise, data scientist can use <code>Jupyter</code> for interactive modeling and submit Kubernetes job through <code>jupyterflow</code> even if they do not know all the details of Kubernetes cluster.

If there is a difference, Zeppelin &amp; Spark has its own job submitting mechanism(<code>spark-submit</code> script) while Jupyter &amp; K8S don’t have. So JupyterFlow came out for closing the distance.

<h3 id="jupyterflow-docs">JupyterFlow Docs</h3>

For more details, please refer to the following JupyterFlow documentations.

<ul>
 <li><a href="https://jupyterflow.com/jupyterhub/">JupyterFlow Installation</a></li>
 <li><a href="https://jupyterflow.com/how-it-works/">JupyterFlow How it works</a></li>
 <li><a href="https://jupyterflow.com/examples/basic/">JupyterFlow Examples</a></li>
 <li><a href="https://jupyterflow.com/configuration/">JupyterFlow Configuration</a></li>
</ul>

<hr>

Now with JupyterFlow, data scientist can write their code, run right away without any worries. How cool is it? Try out JupyterFlow!

<h2 id="wrap-up">Wrap up</h2>

JupyterFlow is my personal open source project. It is still in its early stage of development which is imperfect and has some bugs. However, I believe JupyterFlow has its own great strengh and opportunity to change the ML tool field.
I have found out that there is no other similar appoach like JupyterFlow and the market still has no “De facto standard” ML tools. I am keep investing my spare time to improve this project.
If you are interesting about this project, feel free to contact me by any means.

<ul>
 <li>email: hongkunyoo (at) gmail</li>
 <li><a href="https://github.com/hongkunyoo/jupyterflow/issues/new">Github issue</a></li>
 <li>Blog comments</li>
</ul>

Any comments, questions, request for trouble shooting, feedbacks and joining the project are welcome!

 </section>

 
 
 

 </article>

</div></div>

커피고래의 노트

hongkunyoo

Introducing JupyterFlow, a better way to scale your ML job on Kubernetes.

JupyterFlow - Better way to scale your ML job

댓글