Stories by Park Chansung on Medium

In the age of Large Language Models (LLMs), engaging in conversations with AI has become increasingly common. However, as these interactions grow longer and more complex, keeping track of the key information and insights can become a challenge. This is where the need for a personalized conversation summarization tool arises, helping users distill the essence of their LLM dialogues and retain valuable knowledge.
This project, called Adaptive/Progressive Summarization, aims to address this challenge by providing a unique approach to LLM conversation summarization. Unlike generic summarization methods, our project focuses on creating personalized summaries that reflect the individual user’s interactions and needs. By tailoring the summarization process to each user, we ensure that the generated summaries are not only accurate but also relevant and meaningful to the individual.
One of the key features of our project is the progressive summarization technique. Instead of generating a completely new summary for each interaction, we refine and enhance the existing summary based on the latest conversation. This approach ensures consistency and avoids the problem of disjointed summaries. By incrementally building upon the previous summary, we create a comprehensive and cohesive record of the entire LLM interaction.
This project was done in collaboration with Sayak Paul during the Vertex AI Sprint that happened in early 2025 .
Case Study
At the heart of this project lies the crucial role of prompt engineering. By carefully crafting prompts that guide the LLM toward generating accurate and relevant summaries, we can significantly enhance the quality of the final output. This involves understanding the nuances of LLM behavior and employing effective prompt design techniques to elicit the desired responses.
## System Prompt
Based on the given summary and the last conversation between you (assistant) and me (user), 
update the summary following the below summary guide.
Summary Guide:
* Do not rewrite the entire summary.
* It is allowed to modify the current summary, but do not eliminate.
* Update only the specific portions necessary to reflect new information or changes.
* Only include information we have explicitly discussed. 
* Do not introduce any new information or topics, even if you have prior knowledge.
* Ensure the summary is factually accurate and reflects the nuances of our discussion.
* While being detailed, also aim for conciseness and clarity in the summary.
* Use markdown formatting to make the summary more readable (using bullet points, numbers are preferred).
* Do not seprate sections for previous and updated summaries.
To illustrate the progressive nature of our summarization technique, let’s take a look at a series of screenshots showcasing how the summary evolves as the conversation progresses. These screenshots were taken by ourselves while experimenting with this project on paper s1: Simple test-time scaling. Also, we used Gemini 2.0 Flash for the test.
First summary: what is this paper about?
I wanted to grasp the surface of this paper, hence I simply asked “what is this paper about?”, then the model gave responses, and below is the summary of the first interactions. After reading the entire paper, I found this summary to be well-written and sufficiently detailed.
The paper "s1: Simple test-time scaling" introduces a straightforward approach to improve language model reasoning performance by using additional computation during the testing phase.
Here's a breakdown:
*   **Problem:** The paper tackles the challenge of enhancing reasoning capabilities in language models (LMs) with a focus on simplicity and replicability. It notes that while recent models like OpenAI's o1 demonstrate improved performance through test-time scaling (increasing computation at test time), the methodology isn't public.
*   **Approach:** The authors propose a simple method based on:
    1.  **Data Curation:** They create a small dataset (s1K) of 1,000 reasoning questions with associated traces, meticulously selected based on difficulty, diversity, and quality.
    2.  **Budget Forcing:** A technique to control the computational resources used by the LM during testing. It forcefully terminates or extends the model's reasoning process. The 'Budget Forcing' approach consists of interrupting a model that generates too many thinking tokens or prolonging its thinking duration by appending "Wait" to the model's generation when it wants to stop.
*   **Methodology:**
    *   Supervised fine-tuning of the Qwen2.5-32B-Instruct model on the s1K dataset.
    *   Applying "budget forcing" to the fine-tuned model during testing to control computation.
*   **Results:**
    *   The resulting model (s1-32B) exceeds the o1-preview model's performance on competition-level math questions (MATH and AIME24).
    *   Budget forcing allows for performance extrapolation beyond the model's baseline, enhancing its accuracy on AIME24.
    *   The authors emphasize that using just a thousand samples is enough to generate a strong model that can compete with large-scale closed source models when a 'Budget Forcing' strategy is applied.
*   **Contributions:**
    *   A simple method for creating a sample-efficient reasoning dataset.
    *   A straightforward test-time scaling technique.
    *   A competitive language model called s1-32B.
    *   Ablation studies of dataset creation and test-time scaling methods.
*   **Key Ideas:**
    *   Test-time scaling can significantly improve reasoning in LMs.
    *   Simple methods can be effective and efficient.
    *   Careful data curation is crucial for good performance.
    *   Budget forcing provides a controllable way to manage test-time compute.
In essence, the paper argues that strong reasoning performance and test-time scaling can be achieved through a combination of curated data and controlled test-time computation without complex reinforcement learning techniques.
Second summary: how did the author collect “reasoning traces”?
Collecting “reasoning traces” is trivial task since no major LLM services provide APIs to collect them, hence I wanted to know where they got the reasoning traces. Below is the updated summary based on the second interaction.
It is worth noting that the updated summary is accurate in that it mentions the “Google Gemini Flash Thinking API” when compared to the paper, and that only the part that deals with reasoning traces has been updated, leaving the rest of the summary completely untouched.
.....
    1.  **Data Curation:** .... The reasoning traces were collected using the Google Gemini Flash Thinking API (Google, 2024). They generated a reasoning trace and solution for each question using this API.
.....
Third summary: s1–32B model is fine-tuned to understand “wait” token?
While interacting with LLM, I could understand the role of the “Wait” token, but I was curious about how to apply it. Specifically, I was wondering if I had to fine-tune my local model to understand the role of the Wait token, or if it could be applied to all models without fine-tuning. Again, the only relevant part is updated properly.
.....
    2.  **Budget Forcing:** ... The s1-32B model is fine-tuned to understand the "Wait" token as a signal to continue reasoning when it would normally stop, refining its reasoning and potentially correcting errors.
.....
Fourth summary: what are the limitations of this work?
I understood the core concept and mechanism of the paper, and I wanted to know the limitations. Below are the appended limitations at the end of the summary without hurting other parts, which makes this summary perfectly closed.
......
*   **Limitations:**
    1.  **Performance Plateau:** The effectiveness of budget forcing diminishes after a certain threshold of increased thinking time.
    2.  **Context Window Constraints:** Exceeding the language model's context window due to prolonged reasoning can lead to errors.
    3.  **Limited Extrapolation:** The current algorithm has limitations in achieving substantial extrapolation of test-time computation.
    4.  **Repetitive Loops:** Forcing the model to think longer can cause it to enter unproductive repetitive loops.
    5.  **Generalizability/Task Specificity:** The approach's applicability to different reasoning tasks or domains beyond the curated dataset (s1K) is uncertain.
    6.  **Dataset Size Still Needed:** The approach relies on a curated dataset of 1K samples.
    7.  **Dependency on Pre-trained Model:** The approach depends on fine-tuning an existing pre-trained and instruction-tuned model.
    8.  **Limited Exploration of Parallel Scaling Methods:** The paper primarily focuses on sequential test-time scaling and does not explore parallel methods in depth.
...... The authors also acknowledge limitations in scalability, generalizability and the need for more robust reasoning strategies.
Design of User Interface
To better understand the effectiveness of this approach, tracking how the summary is changing over time is crucial. To this end, we have developed a simple yet powerful Gradio-based application.
As you can see, within the application, you can navigate the history of summaries and compare them. The highlighted texts in green mean addition, while texts in red mean subtraction compared to the previous summary. Also, after a long conversation, all we want to do is to store what we have learnt, and this could be easily done with “Download” button at the bottom.
We have hosted this application on Hugging Face Space, so feel free to try yourself. Also, we are managing the source codes of this project in a GitHub repository. So, please check them out and let us hear any opinions on this approach.
Acknowledgments
This is a project built during the Vertex sprints held by Google’s ML Developer Programs team. We are thankful to be granted good amount of GCP credits to do this project.
Distilling from Dialogues: Finding Meaning in LLM Interactions was originally published in Google Developer Experts on Medium, where people are continuing the conversation by highlighting and responding to this story.

Distilling from Dialogues: Finding Meaning in LLM Interactions

Adapting to changes of data by building MLOps pipeline in Vertex AI

This post shows how to build a machine learning pipeline for object detection tasks. The objective is to demonstrate how to realize MLOps to prepare data drift situations using Vertex AI’s AutoML and Cloud Function.
Motivation
I have a three year old daughter, and I realized that her face is continuously changing so dramatically month by month. Google photo is already doing an excellent job to recognize faces by tracking changes in people’s faces. However, I figured that I could go even further like classifying pictures of her not wearing masks with layered predictions later. That would be a topic for the next article.
Also, I thought this is a great example for MLOps because my wife and I have taken lots of pictures of her just like usual parents. That means I have enough data to build a model, and the data already contains potential data drift since I have pictures of her in the age of infant, one, two, etc.
Building Pipeline and Trigger
We need Vertex AI, Google Cloud Storage, and Cloud Function to build a machine learning pipeline and a trigger. Vertex AI is not a single service, but a combination of lots of different AI related services. To be specific, the project for this post leverages Vertex AI’s Dataset, AutoML, Notebook, Pipeline, Model, and Endpoint features altogether.
Figure 1 shows a step by step instruction on how to build an entire pipeline. The best way to understand the blueprint of a machine learning system is to think about components consisting of the workflow. Let’s go through one by one. Just keep in mind that each component does not represent a standalone job but they are connected altogether in order.
Data Preparation
Data is the heart of machine learning, and we can’t do anything without it. So, there is no doubt that we need to prepare the dataset first. Vertex AI provides a managed dataset feature. You can simply import data from local storage, or you can just point to GCS location if you already have the dataset in an existing GCS bucket.
However, as you know, data itself is not enough. We need labels/annotations. Vertex AI Dataset lets you import labels directly at the time of importing the raw data. You just need to make sure the labels are formed in a suggested manner. You can find how to make your own labeling file here according to data types.
Training
There are multiple options for training a model against your dataset. This post specifically shows how to leverage Vertex AI AutoML features to realize MLOps. Here are three reasons why I have chosen AutoML. First, I do not have to care about modeling. All I have to do is to prepare the dataset in the right format to be recognized by AutoML. Fortunately Vertex AI Dataset perfectly matches to Vertex AI AutoML, so there is no additional workload for this.
Second, when the dataset evolves, we can not guarantee that the current state-of-the-art model is good enough. We probably have to run multiple experiments by writing different versions of codes for data engineering, modeling, and hyper-parameter tuning. AutoML mostly shows the top notch results. That is because the internal algorithm is likely modified and maintained by Google engineers over time, and it potentially guarantees that we almost always leverage the reliable state-of-the-art AutoML modeling techniques.
Last but not least, It is simple to integrate Vertex AI AutoML into Vertex AI Pipeline. Vertex AI Pipeline is simply a wrapper service for Kubeflow Pipeline, and Google has defined a bunch Kubeflow components that are smoothly fused into the standard Kubeflow Pipeline. This means you can leverage Vertex AI AutoML while writing standard Python codes for custom components and connect them.
Deployment and Monitoring
Deployment can be seen as one operation combined with two separate operations, model export and serving an endpoint. Vertex AI supports both of them via Vertex AI Model and Endpoint. Vertex AI Model is a central place where all the trained models are managed along with their versions. You can view the trained results with metrics, test out a simple prediction with it.
Once you think you are ready to deploy the model for the real world users, you can create an endpoint with a chosen model. Actually, Vertex AI Endpoint manages the endpoint of the model in Google Kubernetes Engine. That means you don’t have to care about the scalability of the model. Only a couple of nodes can be served during the early stage of your business, but the number can grow smoothly when your business becomes too big to handle user requests with only a couple of nodes.
Vertex AI Endpoint also provides monitoring functionality for predictions/second, requests/second, latency, and prediction error percentage. You need extra effort to handle concept/data drift issues, but it is sufficient to see if there are errors in prediction requests, if the prediction delays more than expected, if the throughput is not enough with Vertex AI. Vertex AI provides additional monitoring features for tabular model and custom trained model to inspect the model behaviour in depth for now, but it is likely that AutoML models are going to be supported in near future.
Pipeline and Trigger
You can do dataset creation, model training, endpoint instantiation, model deployment individually. However, it is much better to construct a pipeline doing all these jobs in a consistent manner. AutoML likely guarantees for you to have the best model. That means all we have to do is to prepare more data and trigger the pipeline when you witness model performance degradation.
Then what can trigger the pipeline run to learn a new dataset? Certainly, there should be an event listening system to detect the changes in the dataset. This is where Cloud Function comes in. Cloud Function can listen to modification events whenever there are changes to a designated GCS bucket. With this ability, we can simply run the pipeline when we record more data.
Initial Operational Workflow
As an initial phase to demonstrate MLOps, we need a base dataset. As shown in Figure 3, there are multiple steps to create a dataset. First, you need to choose a data type and task type. For this project, I have chosen “Image object detection” under the “IMAGE” category. Second, you can upload images from the local filesystem, or if you have images already been uploaded to a GCS bucket, you can simply choose it.
Also, if you have an extra label file, you can upload it from the same UI as in Figure 3. For this project, I didn’t have any labels, so I just clicked “CONTINUE” after uploading a bunch of images.
Luckily, Vertex AI Dataset provides nice labeling tools right in your browser as shown in Figure 4. With this functionality, I have labeled about 100 images by simply drag and drop the mouse position. After doing this, you have a complete dataset meaning the data and associated labels are stored in the GCS bucket.
Now we are ready to build our initial pipeline with the dataset. We can write pipelining codes directly in terminal or favorite IDEs, but it is often a good idea to run the initial pipeline within a Jupyter Notebook. Because it provides a nice interactive environment, we can edit and experiment by going back and forth of the code. Furthermore, Vertex AI Notebook lets you ignore all the hassles about authorizing process for the GCP since it is already running in the GCP environment.
Figure 5 shows how the workflow goes within a notebook for the initial pipeline run. As the first step, we need to import necessary libraries and set some required variables as shown in the code below.
from google.cloud import aiplatform
from google_cloud_pipeline_components import aiplatform as gcc_aip
from kfp.dsl import pipeline
from kfp.v2 import compiler
from kfp.v2.google.client import AIPlatformClient
PROJECT_ID = “YOUR_GCP_PROJECT_ID”
REGION = “GCP_REGION_TO_RUN_PIPELINE” 
PIPELINE_ROOT = “LOCATION_RUN_METADATA_IS_GOING_TO_BE_STORED”
DATASET_META_PATH = “LOCATION_DATASET_METADATA_IS_STORED”
We need three five libraries under the three package. The two blocks of code examples below show how they are used. The first code block shows how to define a pipeline with three pipeline components in it. Note that @component decorator is used to indicate the function pipeline is where the entire pipeline is defined to Kubeflow Pipeline. We can separate the three components into individual functions with @component decorator and hook them in the pipeline. However, I have put everything in one place to make this sample as simple as possible.
ImageDatasetCreateOp component is to import dataset which we have defined via Vertex AI Dataset. In order to instantiate this component, we need to tell three things which are GCP Project ID, GCS path where the label file is stored, and task type.
@pipeline(name=”my-pipeline”)
def pipeline(project: str = PROJECT_ID):
   ds_op = gcc_aip.ImageDatasetCreateOp(
      project=project,
      display_name=”DATASET_NAME_TO_APPEAR”,
      gcs_source=DATASET_META_PATH,
      import_schema_uri=\
         aiplatform.schema.dataset.ioformat.image.bounding_box,
   )
   training_job_run_op = gcc_aip.AutoMLImageTrainingJobRunOp(
      project=project,
      display_name=”my-daughter-od-training”,
      prediction_type=”object_detection”,
      model_type=”CLOUD”,
      base_model=None,
      dataset=ds_op.outputs[“dataset”],
      model_display_name=”my-daughter-od-model”,
      training_fraction_split=0.6,
      validation_fraction_split=0.2,
      test_fraction_split=0.2,
      budget_milli_node_hours=20000,
   )
   endpoint_op = gcc_aip.ModelDeployOp(
      project=project, model=training_job_run_op.outputs[“model”]
   )
The next component is AutoMLImageTrainingJobRunOp. This is a unified component for every image related task. As you can see, you can specify the specific task type in the prediction_type argument. Also note model_type is set to “CLOUD”. This tells AutoML to figure out what kind of resulting model to produce. For example, you can set model_type differently to “CLOUD_LOW_LATENCY_1” if you want to get a lighter model with low latency. There are multiple choices, so please check out the API document for further information. For this project I just left it as the standard average model.
There are three more arguments to consider in the AutoMLImageTrainingJobRunOp component. You can specify training/validation/test split ratio directly. Although you can specify which images should belong to which dataset during the dataset preparation stage, if you explicitly set them within this component, it will ignore the information and randomly assign data according to the ratios. If you can’t decide carefully the splits by yourself, this is a good way to get started. The buget_milli_node_hours is a constraint when to stop training. Because AutoML can grow the size of a model infinitely if you train forever, you have to make a decision when to stop the training process. Otherwise, you are going to pay lots of money without any much accuracy gain. Lastly, which dataset the AutoML is going to be trained on has to be told, and that is done with the dataset argument. One important thing to know is that the dataset argument sets a connection and a dependency between ImageDatasetCreateOp and AutoMLImageTrainingJobRunOp since the training job has to be executed after the dataset creation operation.
The last component is ModelDeployOp. Despite the name, it has abilities to create an endpoint and deploy the trained model to the endpoint. We can explicitly do the two operations separately, but it is convenient to do them all with one component. All you need to do is specify what model to deploy, and that is set via model argument. And again, this argument sets a connection and a dependency between AutoMLImageTrainingJobRunOp and ModelDeployOp.
compiler.Compiler().compile(
   pipeline_func=pipeline, package_path=PIPELINE_SPEC_PATH
)
api_client = AIPlatformClient(project_id=PROJECT_ID, region=REGION)
response = api_client.create_run_from_job_spec(
 PIPELINE_SPEC_PATH,
 pipeline_root=PIPELINE_ROOT,
 parameter_values={“project”: PROJECT_ID},
)
The pipeline is fully defined, then we have to compile it with the compiler.Compiler().compile method. The role of compiling is to construct a pipeline specification by looking up the pipeline function definition. In the pipeline specification, lots of hidden pieces are recorded such as the job dependencies, which cloud machine type and which container image to be used, and more. By specifying package_path argument, the compiler outputs the pipeline specification in a JSON file. With the JSON file, all you need to do to run the pipeline is to pass the JSON file to the Vertex AI Client’s create_run_from_job_spec method. It is very important for automatic pipeline triggering and reusability.
After sending a request to Vertex AI with the JSON file, you will have the initial pipeline running in the Vertex AI Pipeline. When the pipeline run is finished, you can find out the trained model, the endpoint, the model which is deployed to the endpoint through Vertex AI Model, Endpoint UI Panels respectively.
Observing Unexpected Data Drift
It is trivial to build a simple application to send an image to the endpoint for prediction, but you can easily test your model with testing functionality in Vertex AI Model. I have used that feature since I can easily see the predicted result visually on the GCP console.
In order to test that the model should not be able to detect my daughter’s faces at the age of three as expected, I have given some images to the model as shown in Figure 6. However, I got unexpected results that the model has successfully detected her grown up faces.
I have continued testing the model with the recent images, and I have realized that the COVID-19 pandemic had happened in the last year. I found lots of her images wearing masks, and there were a few images wearing sunglasses. This was the unexpected situation when I initially brainstormed the idea for this project, and I soon saw this situation as the data drift problem. As you can see in Figure 7, the trained model has captured the wrong location and different kids’ faces.
It is possible to run the pipeline manually after collecting and labeling new data. However, it would be much nicer to implement an automatic system to trigger pipeline run whenever we have new data because we have created the JSON specification file.
Figure 8 shows the workflow when data drift occurs. As you can see, we don’t need Vertex AI Notebook anymore, and the pipeline run is not executed directly from the notebook. Instead we can make and deploy a small function to be triggered on Cloud Function, and it listens to an event of changes in a designated GCS bucket. One thing to note is that there is a separate GCS bucket for storing the metadata of the final dataset. While labeling your dataset one by one, the metadata keeps changing frequently, and you don’t want the pipeline to be run every time when it changes. Instead, we can export the finalized metadata file into a separate GCS bucket when we think we are done.
The code block below shows all the codes for the Cloud Function. It is so simple since we already have the pipeline specification JSON file. vertex_ai_pipeline_trigger function gets called whenever there are any changes to any files belonging to a designated GCS bucket. Hence, we need to write a simple filtering conditional statement. The below code ensures to run the pipeline when there are any changes to a file with jsonl extension which is supported by Vertex AI Dataset when exporting.
from kfp.v2.google.client import AIPlatformClient
PROJECT_ID = “YOUR_GCP_PROJECT_ID”
REGION = “GCP_REGION_TO_RUN_PIPELINE” 
PIPELINE_ROOT = “LOCATION_RUN_METADATA_IS_GOING_TO_BE_STORED”
PIPELINE_SPEC_PATH = “LOCATION_PIPELINE_SPEC_IS_STORED”
def vertex_ai_pipeline_trigger(event, context):
   print(‘File: {}’.format(event[‘name’]))
   print(‘Extension: {}’.format(event[‘name’].split(“.”)[-1]))
   if event[‘name’].split(“.”)[-1] == “jsonl”:
      print(“target file extension”)
      api_client = AIPlatformClient(
         project_id=PROJECT_ID, 
         region=REGION
      )
      print(“api_client is successfully instantiated”)
      response = api_client.create_run_from_job_spec(
         PIPELINE_SPEC_PATH,
         pipeline_root=PIPELINE_ROOT,
         parameter_values={“project”: PROJECT_ID},
      )
      print(‘response: {}’.format(response))
Inside the conditional statement, the code to run the pipeline is exactly the same to what we have seen from the notebook section. You might wonder if we need extra authentication processes to access Vertex AI from other GCP services. However, it can be streamlined because Cloud Function and Vertex AI are both GCP services.
gcloud functions deploy YOUR_FUNCTION_NAME \
 — trigger-resource YOUR_TRIGGER_BUCKET_NAME \
 — trigger-event providers/cloud.storage/eventTypes/object.finzlize
After writing a python file containing the code block above, we can deploy it to Cloud Function with the shell command below. This command should be run in the same directory where the python file is located, and “YOUR_FUNCTION_NAME” should match to the actual function name defined in the python file. Please find more about this command in the official document. Also, make sure to include requirements.txt in the same directory with any necessary libraries in it. In this project, google-cloud-aiplatform was included to access Vertex AI APIs.
Updating Dataset to Complement
By writing and deploying the Cloud Function, collecting and labeling more data, and exporting the metadata to the appropriate GCS bucket are everything to get a new model for the updated dataset.
Figure 9 shows that I have included more of my daughter’s images wearing masks. I have also included more of her photos recently taken so that the model could recognize her better.
Final Result
When clicking the exporting metadata button on the top right corner, the Cloud Function gets triggered, and it triggers the pipeline automatically.
Figure 10 shows the final model that I got with the newly collected dataset. As you can see it recognizes the faces of my daughter not only in the age of infant but also more recent times, and it also successfully recognizes her faces wearing masks even when there are other children in the same pictures.
Conclusion
In this post explored how to build a simple but extensible MLOps pipeline with Vertex AI with a practical use case. Vertex AI allows you to prepare your own dataset within a browser. You can play with the codes to interact with Vertex AI APIs with the notebook service. After fixing your code base, you can run the initial pipeline and create the JSON specification file which contains all the details on how to run the pipeline later without the actual code. Finally, you can integrate Cloud Function combined with GCS with the Vertex AI to be run automatically whenever there are any changes in your dataset.
Adapting to changes of data by building MLOps pipeline in Vertex AI was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

Twitter has become the most important platform for various people to get recent news since famous theorist, practitioners, etc. in your interest are continuously posting what they have found, built, and more.
I believe I am not the only one who retweets whenever I encounter interesting tweets. I tell myself like “ok, this looks interesting, but I am not going to read it right now. Let me put it in my feed, so I can read it later on..”. But this never happens.
I have decided to build a small application that pushes a newsletter containing all the retweeted tweets every week. However, if I just send them without any pre-processing, there is no difference from just scrolling around through the Twitter application. This is where Machine Learning might play a great role by classifying tweets in a meaningful way.
Overview
The name of this project is “personalized/curated newsletter service”, but I am not going to give an in-depth explanation in this blog post. Instead, I am going to provide a big picture about what is MLOps(Machine Learning Operations) and how you can realize it within GCP(Google Cloud Platform) environment.
The following components will be covered in this blog post with short descriptions. In-depth explanations are expected to be delivered with dedicated additional blog posts in the future.
GCP Kubernetes Engine: manages underlying hardware resources and allocate appropriate resources to appropriate jobs.
GCP AI Platform (Pipeline): manages machine learning job dependencies by guaranteeing certain jobs should be run beforehand in Kubernetes environment.
TensorFlow Extended (TFX/MLOps): a high-level framework providing a handy way of write job specifications for AI Platform Pipeline and Kubernetes.
GCP AI Platform (Training): a distributed training service at scale with a highly flexible configurability.
GCP AI Platform (Prediction): a scalable model serving service with a unified end point and model version control.
GCP Cloud Storage: a filesystem service that you can save files that are commonly used for machine learning (CSV, Parquet, TFRecord,…)
GCP Cloud Function: a set of logics triggered based on event. You can listen on the changes of data in Cloud Storage and trigger running the machine learning pipeline for the newly collected data.
GCP Cloud Build / GitHub Action: a unit test and deploy service for a newly integrated/merged code base. The machine learning pipeline can be run upon the new codes for different data pre-processing or modeling.
These components can be combined all together to build an entire architecture for my toy project. It’s something like in Fig1. You don’t need to worry if you don’t understand what each component does. I will give short descriptions for each, and you can come back and check if it makes sense after you finished reading this blog post.
“curated newsletter service” sounds easy if you think only about the application part because it’s simply querying to public APIs. However, imagine you are the one who is building your own APIs, and that APIs are built on top of the power of machine learning.
Why Do We Need Machine Learning Operation?
This is better understood with another question “Why do we need to consider different CI/CD pipeline for machine learning project?”
I believe that the simplest answer to this question is “because the data changes over time whether or not you want”. Indeed we are not the main agents but the environment, and the environment makes changes to the data, and we can not stop the changes happening. So it makes sense that we always have to be prepared for such situation. Otherwise, we could take criticism because of the lack of responsibilities.
There are two common scenarios that we, as a developer, have to handle when data changes (Assume that I have collected brand new data and labeled them appropriately).
The currently deployed model re-learn the changed data accordingly. In this case, the system listens to the data change event, and run the whole machine learning pipeline.
The currently deployed model couldn’t achieve the desired accuracy, so we need a new model with appropriate data pre-processing. A new model means the code base has changed. In this case, the system should unit-test the new code base, listens to the code integration, and run the whole machine learning pipeline.
Why GCP and Why Cloud in General?
The whole machine learning pipeline consists of multiple components. Fig2 was introduced in a popular paper “Hidden Technical Debt in Machine Learning Systems” to highlight modeling is just a small part of the whole pipeline.
I disagree with the size of the box for “ML Code”. Modeling is a hard work in my opinion, and I don’t think something is more important than the others. I just want to look at the Fig2 in a way that there are multiple parts that we have to consider when we are planning to roll out machine learning powered products.
The thing is there are several components that we have to hook together. If you have ever experienced to integrate different software components into a complete system, you may know how hard that could go. Especially, building an infrastructure is hard one because data scientists or engineers are normally not familiar with the deep knowledge about hardware, software, operating systems, and so on.
However, infrastructure is very important at the same time since everything should be run at scale. For instance, more of high performance CPUs and high capacity RAMs should be get involved when you have more data. As state-of-the-art(SOTA) models in machine learning world are getting bigger and bigger, the model training system should be able to expand accordingly. You cannot predict when the number of users for your service gets exploded, so the serving system should be able to expand accordingly as well.
It is not easy to do these things at once, even if you are an open source genius. You have to spend quite large amount of time to build such system, and it has to be constantly monitored. In my opinion, the best option instead is to leverage public cloud services and gradually move from cloud to on-premise environment as necessary(when you think you are ready).
Just like other cloud platform, you can consider Google Cloud Platform(GCP) as an all in one solution for your machine learning product. You still need to hook different services provided by GCP, but It is much easier because all of them are in the same platform. Also they are not open sourced project which means much of their functionalities are proven to be working for the most of the cases.
I chose to use GCP not because that I am Machine Learning Google Developers Experts(ML GDE) but because GCP is the best environment when working with TensorFlow, TensorFlow Extended, and other frameworks released by Google. In my opinion, PyTorch is a great option for modeling, but the ecosystem is not mature to support MLOps yet.
Role of Each GCP Services in MLOps
I this section, I am going to briefly overview each GCP services comprising the whole MLOps system in bottom-up approach(like from infrastructure to higher level services). In-depth explanation and hands-on practice will be covered in the future writings in separate blog posts.
Google Kubernetes Engine
Google Kubernetes Engine(GKE) is a fully managed Kubernetes hosted in GCP. Here, “Fully managed” means that you care what it does not how it works under the hood. What GKE does is simple. It manages underlying hardware resources by grouping them into what’s called “Node Pool”. Then GKE picks nodes suitable to requested jobs. For instance, a request could be like “I need a GPU” or “I need a node with more than 10G RAM”.
Within a native GKE or Kubernetes environment, you should write long specification consisting of metadata, labels, docker images, policies, etc. all by yourself. Don’t worry though. You don’t need to handle a very specific request and resource allocation rules by yourself when GKE is combined with TFX and AI Platform Pipeline service that we will discuss shortly. You can just think of them as a high-level wrapper to control GKE in a meaningful way for machine learning specific pipeline.
AI Platform Pipeline
As you know, jobs consisting the whole machine learning pipeline are dependent on other jobs. That means certain jobs must be finished before preceding the next ones. For example, data should be injected first before the data preprocessing is performed, and data preprocessing should be done before the model training, and so on.
Generally speaking, GKE doesn’t care about the order dependency. It just rent out available resources to execute any given jobs. Also, jobs running on GKE are stateless which means the application itself doesn’t store any permanent information, so it is programmer’s duty to manage attached storage for saving permanent information. Also, attached storage is not enough for machine learning pipeline since information should be shared across different jobs.
Kubeflow is the saver for this complicated dependency management. As the name “flow” states, it manages a flow of given jobs. You write some different parts of codes specifying dependencies between jobs, then Kubeflow wraps each job in a container and passes them to GKE. Information sharing is handled internally.
However, Kubeflow itself is very hard to be managed. I guess a whole team or at least a person should have a dedicated role for installing, managing, and monitoring for it. Fortunately GCP also provides fully managed version of Kubeflow, and that is AI Platform Pipeline. With AI Platform Pipeline, you no long need to care the hassles.
AI Platform Notebook
AI Platform Notebook is just a Jupyter Lab hosted on GCP. The only thing that I want to mention about AI Platform Notebook is it can be pretty easily hooked up with AI Platform Pipeline. That means you could easily experiment writing codes for AI Platform Pipeline because notebook is a nice interactive environment.
However, you almost certainly don’t want to write codes for AI Platform Pipeline by yourself because you should have a deep understanding about the internal runtime environment of the AI Platform Pipeline. Of course you want to go deeper and deeper, and you probably have to master the Kubeflow eventually. But you don’t want to spend too much time on it if what you really care is to build up the whole pipeline then inspecting the details later(like you are desperate to launch a product for now).
TensorFlow eXtended(TFX) is the rescuer TFX is just a kind of wrapper framework for defining job specification that could be easily thrown to AI Platform Pipeline. It is like letting you only focus on your business logics while leaving the internal details aside.
TensorFlow eXtended(TFX)
As the name says, TFX is an extension for TensorFlow. TensorFlow is a modeling framework for machine learning and deep learning, and it has nothing to do with production environment handling. TFX comes with lots of add-ons along with TensorFlow to make models written in TensorFlow to be production ready.
Fig6 shows a list of standard TFX components. To name a few with a short description, let me list some of them in the below.
ExampleGen: responsible to handle data injection. Data could be come from local filesystem(with variety of formats like Parquet, CSV, TFRecord,…), BigQuery, Google Cloud Storage, and so on.
Transform: builds up a data transform graph which is going to be attached to the model graph later. It could let you avoid training/serving skew problem. You can simply use TensorFlow library along with TensorFlow Transform(TFT) library.
Trainer: is all about defining and training model. In an early version of TFX, only tf.estimator based models were allowed. However, you can just write tf.keras based models as well which should be very familiar to you. Also, you can easily combine this component with AI Platform Training service for distributed model training.
Pusher: is to push the blessed model to designated location. “Blessed” here means the best one by evaluating by given metrics and comparing with the latest model. Also, you can easily combine this component with AI Platform Prediction service, and if you do, you will be a model version management system and end point for serving the model with a small amount of effort.
Initial Launch for the Pipeline
Actually, you don’t need to run TFX pipeline on a notebook environment. However, it is recommended to do so if it is the initial phase of the project because you have to interactively experiment with TFX, cloud services, cloud environment quite a lot.
Fig7 shows what happens when you run TFX pipeline. Job specifications are tossed to AI Platform Pipeline, and then AI Platform Pipeline wraps each job in a container and pass them to GKE, and then GKE executes each job in a currently available resources. As you could see, each job is guaranteed to be run in a sequential manner.
AI Platform Training/Prediction
GCP comes with a lot more services than just AI Platform Notebook/Pipeline and GKE, and there are some services dedicated for data pre-processing, model training, model tuning, and model prediction. You want to leverage those services because of their scalability with flexible configurability. Also, note that GKE has no idea about distributed training which more than one GPUs are collaborating.
Just imagine the following situations.
Your data grows bigger: You continuously collects more data, and you need more computing resources for data preprocessing accordingly. And for this particular job, it is normal that high-end CPUs and high-capacity RAMs are required.
Your model and data grow bigger: As you collect more data, you can expect to get the most out of the current model much further. Or since SOTA models are announced too often these days, you want to replace the existing model with more promising ones. In either cases, you certainly need more GPUs or even TPUs in a distributed guaranteed environment.
The number of end users grow bigger: It could be enough to have just one server for serving a model end point if you have like 100 of users. Then the number of end users grows to like 100,000. If so, one server is not enough, but you should configure more servers for your model to accommodate the growing number of users.
The listed situation can go in a reverse direction, so the key is the scalability, and the scalability comes from the powerful infrastructure. However, infrastructure is very hard to configure and manage since you should have a solid understanding about hardware, system programs, operating system, software infrastructure like Kubernetes, machine learning in general, and more. There should be a whole team to do this. But, this thing can be super easy if you leverage services provided by GCP.
The listed each situation can be mapped to a dedicated GCP service, and you just need to make a small changes to TFX configuration.
Your data grows bigger: Dataflow
Your model and data grow bigger: AI Platform Training/Tuning
The number of end users grow bigger: AI Platform Prediction
Data Sources
The data for machine learning can be stored in multiple ways, but the most usual cases might be using filesystem(in a number of different formats such as CSV, Parquet, TFRecord,…) or DBMS(Database Management System). In GCP, you can leverage Cloud Storage service for the formal case and BigQuery for the latter case.
Of course, both of them can be easily hooked up with TFX. You just need to import appropriate package(ExampleGen for filesystem, BigQueryExampleGen for BigQuery). Also note that with Cloud Storage service, you could get data version control for free.
Cloud Function & Cloud Build
Now, at this point, the entire machine learning pipeline is established. However, there are two missing parts. We don’t want to run the entire pipeline every time by opening up a notebook. Rather, we might need automatic ways to run the pipeline.
Then, in what situations do we want to run the pipeline automatically?
When we have gathered more data
When we code up better data pre-processing strategies or more recently announced(published) SOTA models.
In the former case, there should be something that is capable of listening on the changes to the data version and trigger actions based on that listening. In the latter case, you should be able to run the pipeline by calling an endpoint. This is the same procedure to what’s done with notebook, but we need an automatic way at this time.
Cloud Function in Fig10 is a serverless service. That means it wakes up on a certain event and run the specified codes. On GCP environment, one of the events that trigger a Cloud Function is the changes in Cloud Storage or BigQuery. So, by simply listening on that event, we could determine when the whole pipeline should be run.
Cloud Build in Fig11 is a source code building service when there are changes in the code repository. You can run a number of unit tests, and when it is good to go, a new machine learning pipeline code can be deployed and run. This is a useful feature when we modify the existing codes to support different data preprocessing strategies, different models, or even different deployment options for like TFJS or TensorFlow Lite.
Summary
We have seen which GCP services could comprise the entire end-to-end machine learning pipeline system.
Hardware and software infrastructures are managed with GKE and AI Platform Pipeline, and job specification to be passed to AI Platform Pipeline could be written in high-level TFX framework. Also, scalable data pre-processing, distributed model training, and scalable model serving can be handled by dedicated GCP services which are Dataflow, AI Platform Training, and AI Platform Prediction. There are a number of storage options, but my recommendation is to use Cloud Storage or BigQuery. When those storage options are combined with Cloud Function, changes in data can be detected, and the pipeline can be triggered to re-train the model with newly collected data. Finally, Cloud Build lets you write/modify new codes and trigger the pipeline based on the newly integrated code base.
However, you can map each GCP service to other equivalent services or open source projects if you still think you are going to have to build everything by yourself.
Who am I?
My name is chansung park, and I am one of Machine Learning Google Developers Experts(ML GDE). Although I am interested in everything in machine learning, my focus in this year(2021) is MLOps especially in GCP environment.
MLOps: Big Picture in GCP was originally published in Google Developers Experts on Medium, where people are continuing the conversation by highlighting and responding to this story.

Stories by Park Chansung on Mediumhttps://medium.com/@parkchansung

Distilling from Dialogues: Finding Meaning in LLM Interactions

Adapting to changes of data by building MLOps pipeline in Vertex AI

Adapting to changes of data by building MLOps pipeline in Vertex AI

MLOps: Big Picture in GCP

Stories by Park Chansung on Medium
https://medium.com/@parkchansung