In the past we have studied how to do limited number of concurrent tasks via buffered channels, an approach inspired from golang. This time we make use of native API of Semaphore to achieve the same goal.
We study how to embed value object into JPA entity class definition, and study how to auto-map our value object back to plain DTO object for the frontend
Certain resources are not supposed to be recreated even we recreate our infrastructure via terraform. Let's study how we define and import database and s3 bucket separately.
It is very common to have lamdba function being called by another function. This time we study two kinds of lambda functions to be invoked, one is snapstarted springboot lambda function, another one is an ordinary console lambda function.
Creation of modal is always a tedious task. We can create modal easily by ant-design's Modal component. But improper implementation can lead to plenty of highly non-reusable modals as the common mistake is to create that modal at the component we use it, making it tightly coupled with the state at which the modal is defined.
When business grows, a domain object becomes very bulky if all behaviour and the corresponding validation are added to the same entity file. We study code separation by the language feature of Kotlin using function literal with receiver.
We discuss how to bring the power of caching/debouncing, the cahcing and cache-invalidatin, and also the handy booleans like isLoading of react query into world of react thunk actions.
After real project experience with AWS cloud, I revisit the knowledge in terraform in order to replicate the cloud infrastructure from UAT to PROD effectively.
Let's discuss how to orchistrate the interaction of lambda functions which are network-isolated, especially how to endow lambda functions with security group.
Suppose we want to test production-like environment by making a production build for mobile, it is helpful to host multiple backends in localhost and expose it via https, multiple ngroks can do the trick for us.
Setting eslint is not enough as people can still ignore it and commit their code. We introduce husky which provides a pre-commit hook to prevent unpassed code from being pushed.
Due to the lambda-nature, snapStarted spring boot differs from ordinary spring boot application as it lacks the write permission to the /var/task of the lambda function.
In testing we usually make use of in-memory database to separate the test data with our actual working database. For a long time people use H2 in spring boot for that purpose but that database is not eventually what the production uses, now we can align both!
Installing python and executing a python script with argument can be tedious to teammates who are not used to python. The same siutation can apply to all other languages, let's simplify the execution by doing it on github actions.
Working with existing database via jpa amounts to the need to manually model the table by entity classes. We study how to modify the data class generated by JOOQ into a "pre-entity" open class that can be inherited to define domain model.
We have talked about monolithic DDD in the context of spring boot with the help of spring-provided ApplicationEventPublisher instance via dependency injection. This is not unique to spring and let's bring it into nodejs in the express framework.
A Stripe event is always delayed, and some system relies on database persistent change by Stripe events. We study how frontend can wait for the event before fetching latest data.
Traditionally lambda function is as simple as running the function defined in a zipped package. But when dependencies get complicated and when file size inevitably exceeds 250MB limit when being unzipped, we need to consider using docker image as an alternative.
When creating a transaction we need to wait for the checkout-session-completed event in order to execute follow-up action in our own system, we relies on metadata to follow the transaction.
The generation of the bean of WebMvcConfigurer suffices to provide a global configuration of CORS via Rest Controller, but that fails to configure non-mvc requests (such as those intercepted by our interceptors). Let's study an alternative.
Sometimes it is much more convenient to save multiple config files in a repository than to git-ignore it. We study the encrpytion of those files using git-crypt.
Study a simple query function WIDTH_BUCKET to obtain a set of time-series data and study nice chart library to generate a bar-chart with multi-columns.
This is a study on gradle to help understand how to manage a project by modules. In java this helps separate the dependencies precisely, i.e., no one can access resource from incorrect layer. This makes creating unit tests much more easily (in case you have written queries directly in controller layer, you know what I mean).
In the past we have studied CDC with the help of confluent here. This time we host everything locally to prepare ourselves to host CDC without the dependency on confluent.
Sometimes a library may require user installing gcc, c++, ca-certificates, or some other linux-specific libraries. Let's use docker image to provide us a consistent working environment.
The concept of message queue is easy but writing them can easily be messy due to boilerplate code, we summarize how to organize them into a MessageQueue class.
The default Datetime object in PGSQL by default is UTC+0. Application-wise we wish an absolute timestamp that is compatible with frontend (i.e., javascript), here is how.
We record a vscode configuration to customize where to save our image when pasting image in md files with automatically generated url inside markdown file.
Methods of sharing environment variabls has always been being diversified. Some may just keep the .env in the project. Some may share it via instant messagers. Some may store it in a webpage that require a login. We share a method that should be a norm for intense aws users!
We record how to make use of the native gmail api to send email without sendgrid which cost money with only 200 mails a date in free plan (while gmail api is free).
In the past we have discussed SSE by kind of short polling in the backend (keep looping to see whether a key has message to pop out in redis queue). This time we send messages to frontend by listening subscriptions on EventEmitter, an approach very native to languages in which channel is implemented, like go and rust.
In the browser our route consist of many information, like messageId, emailId, etc, which enables our web page to select correct data based on the route.
In this post we discuss how to effectively construct such url with type safty under a RouteSchema.
An introduction to how to effectively create SSE event mimicing the single-thread event loop adopted by nodejs in springboot, and how do we actually implement it in nodejs as well.
I used to use useRouteMatch in react-router-dom v5, which has been changed completely in v6 into something called useMatch or useMatches. We talk about building the param searching function on our own to get rid of these unhandy "black boxes".
Record the mongo query using aggregation pipeline to get the results as in the title in my usual work. These include $lookup, $project, $arrayElemAt, {$gt: {$size: 1}}, etc interesting operations.
In the course of translating pytorch model into libtorch model there are traps and tricks that are worthing being recorded. Also record the simple use of opencv as it substitutes the role of numpy in python.
Sometimes the constant CMAKE_CXX_STANDARD does not guarantee the C++ standard we use in compilation. We add a line to guarantee which target is compiled in which C++ standard.
We can avoid heap memory allocation by using C style array stored in stack memory, given that we don't need such a flexibility of dynamic array (the std::vector class) and the size of the target array is known beforehand.
In practice we can directly use torchvision.models.swin_t to use pretrained model. The Reimplementation of the model is for sheer purpose of my self-learning.
Continuation of the previous blog post on transformer, discuss more modern pipeline for training a transformer (or any NLP task in general) using the latest torchtext.
Sometimes a loss becomes nan in rare occasion (e.g., it occurs once per 5~7 epochs), in this case we record a script to restart the training using the latest weight.
Break down the code for my own study. e.g., the way it does image cropping, resizing, etc augmentations while keeping the vertice consistent are very valuable reference, the way it does nms is also succint! Source Code:Link
Mask in numpy, tensorflow and pytorch is very useful to filter out desired values, but sometimes the index that the mask marks as True is also import, we record a function to retrive those information.
I made this study note on 5th of March, 2021: 2021-02-28-yolov3-algorithm-drilling in pdf format. For a refresh of memory I reformat the content into an md file for this blog for easy reading and reference.
Discrete Cosine Transform (DCT) is not only used in image processing, it is also used in signal processing of sound such as computing MFCC coefficients as a feature vector.
Completed the Rust official tutorial from its online book and concluded the 20 chapters by a multithreading simple web server. Try to record the subtitle detail for future reference.
From time to time it is easy to forget the formula to calculate the output shape of Conv2D, MaxPooling2D, etc, layers. Record some sample code to test shapes easily.
Logging is undoubtedly the most important part of an application. We study how to create a logger that tells us a message comes from which file and which line.
In the past I have learnt the most basic GAN and DCGAN using pytorch, but I am more familiar with tensorflow. I attempt to understand gradient tape using by using these two GAN again with mnist dataset.
Sometimes we would like to call python script inside our node js program. We discuss how to execute a command, how to read the log and error in node-js and how to wait until this process is done.
I am working on a desktop application by using Electron + Next.js in typescript. Since I cannot live without debugger, I have spent time searching debug config in the internet and finally come into a functioning configuration!
Sometimes it is convenient to directly execute a test file to understand how the project work. In the past we discussed how to make a runnable test file in python, in javascript we can use exactly the same approach for debugging.
When a class grows to 500 to 1000 lines of codes it becomes hard and tedious to maintain because there starts to be many class methods, some are short, some are huge. It is a good starting point to separate particularly long functions from the class with very very minimal effort, but how?
We list some potential problem of using shallow copy and how to avoid them by deep copy. We can also improve computation efficiency when a variable is never reused but needed to be passed into a function/class attribute. We achieve this by using move constructor.
In classification tasks there are already state-of-the-art models trained from a myriad of images. We try to make a network surgery on one of them (VGG-16 this time) to quickly classifiy our custom dataset with good result.
Digest the material from Convex Analysis by Gert Wanka. I modify the proofs by quite a bit once there is available theorem in my mind that is not presented in the text or I have alternative proofs from other material.
Start the journey of convex analysis in an attempt to fully understand the technical tools needed in optimal transport. In this article we digest the material from Chapter 5 Convex Optimization in Function Space, I also fill in the important missing detail in the proof (the author is a bit rushing).
In backend we can loop through the files inside a directory using fs.readdir, we introduce a function that can achive the same thing in frontend using webpack's require.context function.
Study the formulas on conditioned joint-distribution such as relation between , and . We also discuss the latent variable model in variational Auto Encoder.
1importcreateCachefrom"@emotion/cache";23const isBrowser =typeofdocument!=="undefined";45// On the client side, Create a meta tag at the top of the <head> and set it as insertionPoint.6// This assures that MUI styles are loaded first.7// It allows developers to easily override MUI styles with other styling solutions, like CSS modules.8exportdefaultfunctioncreateEmotionCache(){9let insertionPoint;1011if(isBrowser){12const emotionInsertionPoint =document.querySelector(13'meta[name="emotion-insertion-point"]'14);15 insertionPoint = emotionInsertionPoint ??undefined;16}1718returncreateCache({key:"mui-style", insertionPoint });19}
pages/_document.js
1import*asReactfrom"react";2importDocument,{Html,Head,Main,NextScript}from"next/document";3importcreateEmotionServerfrom"@emotion/server/create-instance";4importcreateEmotionCachefrom"../src/createEmotionCache";56exportdefaultclassMyDocumentextendsDocument{7render(){8return(9<Html lang="en">10<Head>11{/* PWA primary color */}12<link rel="shortcut icon" href="/favicon.ico"/>13<meta name="emotion-insertion-point" content=""/>14{this.props.emotionStyleTags}15</Head>16<body>17<Main/>18<NextScript/>19</body>20</Html>21);22}23}2425// `getInitialProps` belongs to `_document` (instead of `_app`),26// it's compatible with static-site generation (SSG).27MyDocument.getInitialProps=async(ctx)=>{28// Resolution order29//30// On the server:31// 1. app.getInitialProps32// 2. page.getInitialProps33// 3. document.getInitialProps34// 4. app.render35// 5. page.render36// 6. document.render37//38// On the server with error:39// 1. document.getInitialProps40// 2. app.render41// 3. page.render42// 4. document.render43//44// On the client45// 1. app.getInitialProps46// 2. page.getInitialProps47// 3. app.render48// 4. page.render4950const originalRenderPage = ctx.renderPage;5152// You can consider sharing the same Emotion cache between all the SSR requests to speed up performance.53// However, be aware that it can have global side effects.54const cache =createEmotionCache();55const{ extractCriticalToChunks }=createEmotionServer(cache);5657 ctx.renderPage=()=>58originalRenderPage({59enhanceApp:(App)=>60functionEnhanceApp(props){61return<App emotionCache={cache}{...props}/>;62},63});6465const initialProps =awaitDocument.getInitialProps(ctx);66// This is important. It prevents Emotion to render invalid HTML.67// See https://github.com/mui/material-ui/issues/26561#issuecomment-85528615368const emotionStyles =extractCriticalToChunks(initialProps.html);69const emotionStyleTags = emotionStyles.styles.map((style)=>(70<style
71 data-emotion={`${style.key}${style.ids.join(" ")}`}72 key={style.key}73// eslint-disable-next-line react/no-danger74 dangerouslySetInnerHTML={{__html: style.css}}75/>76));7778return{79...initialProps,80 emotionStyleTags,81};82};