In this article I will show how we can extract music sources: bass, drums, vocals and other accompaniments using neural networks.
Before you will continue reading please watch short introduction:
Separation of individual instruments from arranged music is another area where machine learning
algorithms could help. Demucs solves this problem using neural networks.
The trained model (https://arxiv.org/pdf/1909.01174v1.pdf) use U-NET architecture which contains two parts encoder and decoder.
On the encoder input we put the original track and after processing we get bass, drums, vocals and other accompaniments at the decoder output.
The encoder, is connected to the decoder, through additional LSTM layer,
as well as residual connections between subsequent layers.
Ok, we have neural network architecture but what about the training data ?
This is another difficulty which can be handled by the unlabeled data remixing pipeline.
We start with another classifier, which can find the parts of music, which do not contain the specific instruments, for example drums.
Then, we mix it with well known drums signal, and separate the tracks using the model.
Now we can compare, the separation results, with known drums track and mixture of other instruments.
According to this, we can calculate the loss (L1 loss), and use it during the training.
Additionally, we set different loss weights, for known track and the other.
The whole UI is kept in the docker image thus you can simply try it:
#for CPU
docker run --name aiaudioseparation -it -p 8000:8000 -v $(pwd)/checkpoints:/root/.cache/torch/hub/checkpoints --rm qooba/aimusicseparation
#for GPU
docker run --name aiaudioseparation --gpus all -it -p 8000:8000 -v $(pwd)/checkpoints:/root/.cache/torch/hub/checkpoints --rm qooba/aimusicseparation
In this article I will show how we can prepare and perform calculations on quantum computers
using OpenFermion, Cirq and PySCF.
Before you will continue reading please watch short introduction:
Currently, there are many supercomputing centers, where we can run complicated simulations.
However, there are still problems that are beyond the capabilities of classical computers,
which can be addressed by quantum computers.
Quantum chemistry and materials science problems which that are described by the laws of
quantum mechanics can be mapped to the quantum computers and projected to qubits.
OpenFermion is the library which can help to perform such calculations on a quantum computer.
Additionally we will use the PySCF package which will help to perform initial structure optimization (if you are interested in PySCF package I have shared the example DFT based band structure calculation of the single layer graphene structure pyscf_graphene.ipynb).
In our example we will investigate H_2 molecule for simplicity. We will use the PySCF package to find optimal bond length of the molecule.
Thanks to the OpenFermion-PySCF plugin we can smoothly use the molecule initial state obtained from PySCF package run in OpenFermion library (openfermionpyscf_h2.ipynb).
Using one of this methods we get optimized quantum circuit.
In our case the quantum cirquit for H_2 system will be represented by 4 qubits and operations that act on them (moment is collection of operations that act at the same abstract time slice).
Before you will continue reading please watch short introduction:
In the last article (TinyML with Arduino) I have shown the example TinyML model which will classify
jelly bears using RGB sensor.
The next step, will be to build a process that will simplify, the model versions management, and the deployment.
Now we are ready to run the MLflow project using command:
mlflow run https://git_repository.git#path --no-conda --experiment-name="arduino"
The model is saved in the MLflow registry and the model version is associated with
the git commit version.
The MLflow model contains additional artifacts:
* artifacts.ino – the arduino code which loads and uses the model
* model.h – the Tensorflow Lite model encoded to hex
* reduirements.ino.txt – the list of Arduino dependencies required by the arduino code
where:
* –device=/dev/ttyACM0 – is arduino device connected using USB
* AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY – are minio credentials
* MLFLOW_S3_ENDPOINT_URL – is minio url
* MLFLOW_TRACKING_URI – is mlflow url
* ${RUN_ID} – is run id of model saved in MLflow registry
Additionally we have several command line options:
ARDUINO MLOPS
Syntax: docker run -it qooba/tinyml-arduino:mlops -h [-r MLFLOW_RUN_ID] [-s ARDUINO_SERIAL] [-c ARDUINO_CORE] [-m ARDUINO_MODEL]
options:
-h|--help Print help
-r|--run MLflow run id
-s|--serial Arduino device serial (default: /dev/ttyACM0)
-c|--core Arduino core (default: arduino:mbed_nano)
-m|--model Arduino model (default: arduino:mbed_nano:nano33ble)
After running the code the docker image qooba/tinyml-arduino:mlops
will fetch the model for indicated RUN_ID from MLFlow.
Then it will install required dependencies using the file requirements.ino.txt.
It will compile the model and the Arduino code.
And finally upload it to the device.
Thanks to this, we can more easily manage subsequent versions of models, and automate the deployment process.
In this article I will show how to build Tensorflow Lite based jelly bears classifier using Arduino Nano 33 BLE Sense.
Before you will continue reading please watch short introduction:
Currently a machine learning solution can be deployed not only on very powerful machines containing GPU cards but also on a really small devices. Of course such a devices has a some limitation eg. memory etc. To deploy ML model we need to prepare it. The Tensorflow framework allows you to convert neural networks to Tensorflow Lite which can be installed on the edge devices eg. Arduino Nano.
Arduino Nano 33 BLE Sense is equipped with many sensors that allow for the implementation of many projects eg.:
* Digital microphone
* Digital proximity, ambient light, RGB and gesture sensor
* 3D magnetometer, 3D accelerometer, 3D gyroscope
* Capacitive digital sensor for relative humidity and temperature
Examples which I have used in this project can be found here.
To simplify device usage I have build Arduino Lab project where you can test and investigate listed sensors directly on the web browser.
The project dependencies are packed into docker image to simplify usage.
Before you start the project you will need to connect Arduino through USB (the Arduino will communicate with docker container through /dev/ttyACM0)
git clone https://github.com/qooba/tinyml-arduino.git
cd tinyml-arduino
./run.server.sh
# in another terminal tab
./run.nginx.sh
# go inside server container
docker exec -it arduino /bin/bash
./start.sh
For each sensor type you can click Prepare button which will build and deploy appropriate Arduino code.
NOTE:
Sometimes you will have to deploy to arduino manually to do this you will need to
go to arduino container
docker exec -it arduino /bin/bash
cd /arduino
make rgb
Here you have complete Makefile with all types of implemented sensors.
You can start observations using Watch button.
Now we will build TinyML solution.
In the first step we will capture training data:
The training data will be saved in the csv format. You will need to repeat the proces for each class you will detect.
Captured data will be uploaded to the Colab Notebook.
Here I fully base on the project Fruit identification using Arduino and TensorFlow.
In the notebook we train the model using Tensorflow then convert it to Tensorflow Lite and finally encode to hex format (model.h header file) which is readable by Arduino.
Now we compile and upload model.h header file using drag and drop mechanism.
Finally we can classify the jelly bears by the color:
In this article I will show how to prepare complete MLOPS solution based on the Feast feature store and MLflow platform.
Before you will continue reading please watch short introduction:
The whole solution will be deployed on the kubernetes (mlflow_feast.yaml).
We will use:
* Feast – as a Feature Store
* MLflow – as model repository
* Minio – as a S3 storage
* Jupyter notebook – as a workspace
* Redis – for a online features store
To better visualize the whole process we will use the Propensity to buy example where I base on the Kaggle examples and data.
We start in Jupyter Notebook where we prepare Feast feature store schema which is kept in S3.
We can simply inspect the Feast schema in Jupyter Notebook:
from feast import FeatureStore
from IPython.core.display import display, HTML
import json
from json2html import *
import warnings
warnings.filterwarnings('ignore')
class FeastSchema:
def __init__(self, repo_path: str):
self.store = FeatureStore(repo_path=repo_path)
def show_schema(self, skip_meta: bool= False):
feast_schema=self.__project_show_schema(skip_meta)
display(HTML(json2html.convert(json = feast_schema)))
def show_table_schema(self, table: str, skip_meta: bool= False):
feasture_tables_dictionary=self.__project_show_schema(skip_meta)
display(HTML(json2html.convert(json = {table:feasture_tables_dictionary[table]})))
def __project_show_schema(self, skip_meta: bool= False):
entities_dictionary={}
feast_entities=self.store.list_entities()
for entity in feast_entities:
entity_dictionary=entity.to_dict()
entity_spec=entity_dictionary['spec']
entities_dictionary[entity_spec['name']]=entity_spec
feasture_tables_dictionary={}
feast_feature_tables=self.store.list_feature_views()
for feature_table in feast_feature_tables:
feature_table_dict=json.loads(str(feature_table))
feature_table_spec=feature_table_dict['spec']
feature_table_name=feature_table_spec['name']
feature_table_spec.pop('name',None)
if 'entities' in feature_table_spec:
feature_table_entities=[]
for entity in feature_table_spec['entities']:
feature_table_entities.append(entities_dictionary[entity])
feature_table_spec['entities']=feature_table_entities
if not skip_meta:
feature_table_spec['meta']=feature_table_dict['meta']
else:
feature_table_spec.pop('input',None)
feature_table_spec.pop('ttl',None)
feature_table_spec.pop('online',None)
feasture_tables_dictionary[feature_table_name]=feature_table_spec
return feasture_tables_dictionary
FeastSchema(".").show_schema()
#FeastSchema(".").show_schema(skip_meta=True)
#FeastSchema(".").show_table_schema('driver_hourly_stats')
#FeastSchema().show_tables()
In our case we store the data in Apache Parquet files in S3 bucket.
Using the Feast we can fetch the historical features and train the model using Scikit-learn library
Using MLflow we can simply deploy model as a microservice in k8s.
In our case we want to deploy the model models:/propensity_model/Production
which is currently assigned for Production. During start the MLflow will automatically fetch the proper model from S3:
In this article I will show how to process streams with Apache Flink and MLflow model
Before you will continue reading please watch short introduction:
Apache Flink allows for an efficient and scalable way of processing streams. It is a distributed processing engine which supports multiple sources like: Kafka, NiFi and many others
(if we need custom, we can create them ourselves).
Apache Flink also provides the framework for defining streams operations in languages like:
Java, Scala, Python and SQL.
To simplify the such definitions we can use Jupyter Notebook as a interface. Of course we can write in Python using PyFlink library but we can make it even easier using writing jupyter notebook extension (“magic words”).
Using Flink extension (magic.ipynb) we can simply use Flink SQL sql syntax directly in Jupyter Notebook.
To use the extesnions we need to load it:
%reload_ext flinkmagic
Then we need to initialize the Flink StreamEnvironment:
%flink_init_stream_env
Now we can use the SQL code for example:
FileSystem connector:
%%flink_execute_sql
CREATE TABLE MySinkTable (
word varchar,
cnt bigint) WITH (
'connector.type' = 'filesystem',
'format.type' = 'csv',
'connector.path' = '/opt/flink/notebooks/data/word_count_output1')
The magic keyword will automatically execute SQL in existing StreamingEnvironment.
Now we can apply the Machine Learning model. In plain Flink we can use UDF function defined
in python but we will use MLflow model which wraps the ML frameworks (like PyTorch, Tensorflow, Scikit-learn etc.). Because MLflow expose homogeneous interface we can
create another “jupyter magic” which will automatically load MLflow model as a Flink function.
%%flink_sql_query
SELECT word as smstext, SPAM_CLASSIFIER(word) as smstype FROM MySourceKafkaTable
which in our case will fetch kafka events and classify it using MLflow spam classifier. The
results will be displayed in the realtime in the Jupyter Notebook as a events DataFrame.
If we want we can simply use other python libraries (like matplotlib and others) to create
graphical representation of the results eg. pie chart.
You can also use docker image: qooba/flink:dev to test and run notebooks inside.
Please check the run.sh
where you have all components (Kafka, MySQL, Jupyter with Flink, MLflow repository).
In this article I will show how to use artificial intelligence to add motion to the images and photos.
Before you will continue reading please watch short introduction:
Face reenactment
To bring photos to life we can use the face reenactment algorithm designed to transfer the facial movements in the video to another image.
In this project I have used github implementation: https://github.com/AliaksandrSiarohin/first-order-model. Where the extensive description of the neural network architecture can be found in this paper. The solution contains of two parts: motion module and generation module.
The motion module at the first stage extracts the key points from the source and target image. In fact in the solution we assume that reference image which we can to the source and target image exists and at the first stage the transformations from reference image to source (T_{S \leftarrow R} (p_k)) and target (T_{T \leftarrow R} (p_k)) image is calculated respectively. Then the first order Taylor expansions \frac{d}{dp}T_{S \leftarrow R} (p)| {p=p_k} and \frac{d}{dp}T_{T \leftarrow R} (p)| {p=p_k} is used to calculate dense motion field.
The generation module use calculated dense motion field and source image to generate new image that will resemble target image.
The whole solution is packed into docker image thus we can simply reproduce the results using command:
NOTE: additional volumes (torch_models and checkpoints) are mount because during first run the trained neural networks are downloaded.
To reproduce the results we need to provide two files motion video and source image. In above example I put them into test directory and mount it into docker container (-v $(pwd)/test:/ai/test) to use them into it.
Below you have all command line options:
usage: prepare.py [-h] [--config CONFIG] [--checkpoint CHECKPOINT]
[--source_image SOURCE_IMAGE]
[--driving_video DRIVING_VIDEO] [--crop_image]
[--crop_image_padding CROP_IMAGE_PADDING [CROP_IMAGE_PADDING ...]]
[--crop_video] [--output OUTPUT] [--relative]
[--no-relative] [--adapt_scale] [--no-adapt_scale]
[--find_best_frame] [--best_frame BEST_FRAME] [--cpu]
first-order-model
optional arguments:
-h, --help show this help message and exit
--config CONFIG path to config
--checkpoint CHECKPOINT
path to checkpoint to restore
--source_image SOURCE_IMAGE
source image
--driving_video DRIVING_VIDEO
driving video
--crop_image, -ci autocrop image
--crop_image_padding CROP_IMAGE_PADDING [CROP_IMAGE_PADDING ...], -cip CROP_IMAGE_PADDING [CROP_IMAGE_PADDING ...]
autocrop image paddings left, upper, right, lower
--crop_video, -cv autocrop video
--output OUTPUT output video
--relative use relative or absolute keypoint coordinates
--no-relative don't use relative or absolute keypoint coordinates
--adapt_scale adapt movement scale based on convex hull of keypoints
--no-adapt_scale no adapt movement scale based on convex hull of
keypoints
--find_best_frame Generate from the frame that is the most alligned with
source. (Only for faces, requires face_aligment lib)
--best_frame BEST_FRAME
Set frame to start from.
--cpu cpu mode.
The GaN network consists of two parts of the Generator whose task is to generate the image from random input and a discriminator that checks if the generated image is realistic.
During training, the networks compete with each other, the generator tries to generate better and better images
and thereby deceive the Discriminator. On the other hand, the Discriminator learns to distinguish between real and generated photos.
To train the discriminator, we use both real photos and those generated by the generator.
Finally, we can achieve the following results using DCGAN network.
As you can see some faces look realistic while some are distorted, additionally the network can only generate low resolution images.
We can achieve much better results using the StyleGaN (arxiv article) network, which, among other things, differs in that the next layers of the network are progressively added during training.
I generated the images using pretrained networks and the effect is really amazing.
In this article I will show how to improve the quality of blurred face images using
artificial intelligence. For this purpose I will use neural networks and FastAI library (ver. 1)
Before you will continue reading please watch short introduction:
I have based o lot on the fastai course thus I definitely recommend to go through it.
Data
To train neural network how to rebuild the face images we need to provide the
faces dataset which will show how low quality and blurred images should be reconstructed.
Thus we need pairs of low and high quality images.
We will treat the original images as a high resolution data and rescale them
to prepare low resolution input:
import fastai
from fastai.vision import *
from fastai.callbacks import *
from fastai.utils.mem import *
from torchvision.models import vgg16_bn
from pathlib import Path
path = Path('/opt/notebooks/faces')
path_hr = path/'high_resolution'
path_lr = path/'small-96'
il = ImageList.from_folder(path_hr)
def resize_one(fn, i, path, size):
dest = path/fn.relative_to(path_hr)
dest.parent.mkdir(parents=True, exist_ok=True)
img = PIL.Image.open(fn)
targ_sz = resize_to(img, size, use_min=True)
img = img.resize(targ_sz, resample=PIL.Image.BILINEAR).convert('RGB')
img.save(dest, quality=60)
sets = [(path_lr, 96)]
for p,size in sets:
if not p.exists():
print(f"resizing to {size} into {p}")
parallel(partial(resize_one, path=p, size=size), il.items)
Now we can create data bunch for training:
bs,size=32,128
arch = models.resnet34
src = ImageImageList.from_folder(path_lr).split_by_rand_pct(0.1, seed=42)
def get_data(bs,size):
data = (src.label_from_func(lambda x: path_hr/x.name)
.transform(get_transforms(max_zoom=2.), size=size, tfm_y=True)
.databunch(bs=bs,num_workers=0).normalize(imagenet_stats, do_y=True))
data.c = 3
return data
data = get_data(bs,size)
Training
In this solution we will use a neural network with UNET architecture.
The UNET neural network contains two parts Encoder and Decoder which are used to reconstruct the face image.
During the first stage Encoder fetch the input, extracts and aggregates the image features. At each stage the features maps are donwsampled.
Then Decoder uses extracted features and tries to rebuild the image upsampling it at each decoding stage. Finally we get regenerated images.
Additionally we need to define the Loss Function which will tell the model if the image was rebuilt correctly and allow to train the model.
To do this we will use additional neural network VGG-16. We will put Generated image and Original image (which is our target) to the network input. Then will compare the features extracted for both images at selected layers and according to this calculated the loss.
Finally we will use Adam optmizer to minimize the loss and achieve better result.
def gram_matrix(x):
n,c,h,w = x.size()
x = x.view(n, c, -1)
return (x @ x.transpose(1,2))/(c*h*w)
base_loss = F.l1_loss
vgg_m = vgg16_bn(True).features.cuda().eval()
requires_grad(vgg_m, False)
blocks = [i-1 for i,o in enumerate(children(vgg_m)) if isinstance(o,nn.MaxPool2d)]
class FeatureLoss(nn.Module):
def __init__(self, m_feat, layer_ids, layer_wgts):
super().__init__()
self.m_feat = m_feat
self.loss_features = [self.m_feat[i] for i in layer_ids]
self.hooks = hook_outputs(self.loss_features, detach=False)
self.wgts = layer_wgts
self.metric_names = ['pixel',] + [f'feat_{i}' for i in range(len(layer_ids))
] + [f'gram_{i}' for i in range(len(layer_ids))]
def make_features(self, x, clone=False):
self.m_feat(x)
return [(o.clone() if clone else o) for o in self.hooks.stored]
def forward(self, input, target):
out_feat = self.make_features(target, clone=True)
in_feat = self.make_features(input)
self.feat_losses = [base_loss(input,target)]
self.feat_losses += [base_loss(f_in, f_out)*w
for f_in, f_out, w in zip(in_feat, out_feat, self.wgts)]
self.feat_losses += [base_loss(gram_matrix(f_in), gram_matrix(f_out))*w**2 * 5e3
for f_in, f_out, w in zip(in_feat, out_feat, self.wgts)]
self.metrics = dict(zip(self.metric_names, self.feat_losses))
return sum(self.feat_losses)
def __del__(self): self.hooks.remove()
feat_loss = FeatureLoss(vgg_m, blocks[2:5], [5,15,2])
learn = unet_learner(data, arch, wd=wd, loss_func=feat_loss, callback_fns=LossMetrics,
blur=True, norm_type=NormType.Weight)
Results
After training we can use the model to regenerate the images:
Application
Finally we can export the model and create the drag and drop application which fix the face images in web application.
The whole solution is packed into docker images thus you can simply start it using commands:
# with GPU
docker run -d --gpus all --rm -p 8000:8000 --name aiunblur qooba/aiunblur
# without GPU
docker run -d --rm -p 8000:8000 --name aiunblur qooba/aiunblur
To use GPU additional nvidia drivers (included in the NVIDIA CUDA Toolkit) are needed.
The popularity of drones and the area of their application is becoming greater each year.
In this article I will show how to programmatically control Tello Ryze drone, capture camera video and detect objects using Tensorflow. I have packed the whole solution into docker images (the backend and Web App UI are in separate images) thus you can simply run it.
Before you will continue reading please watch short introduction:
Architecture
The application will use two network interfaces.
The first will be used by the python backend to connect the the Tello wifi to send the commands and capture video stream. In the backend layer I have used the DJITelloPy library which covers all required tello move commands and video stream capture.
To efficiently show the video stream in the browser I have used the WebRTC protocol and aiortc library. Finally I have used the Tensorflow 2.0 object detection with pretrained SSD ResNet50 model.
The second network interface will be used to expose the Web Vue application.
I have used nginx to serve the frontend application
Application
Using Web interface you can control the Tello movement where you can:
* start video stream
* stop video stream
* takeoff – which starts Tello flight
* land
* up
* down
* rotate left
* rotate right
* forward
* backward
* left
* right
In addition using draw detection switch you can turn on/off the detection boxes on the captured video stream (however this introduces a delay in the video thus it is turned off by default). Additionally I send the list of detected classes through web sockets which are also displayed.
As mentioned before I have used the pretrained model thus It is good idea to train your own model to get better results for narrower and more specific class of objects.
Finally the whole solution is packed into docker images thus you can simply start it using commands:
To use GPU additional nvidia drivers (included in the NVIDIA CUDA Toolkit) are needed.
We use cookies to ensure that we give you the best experience on our website. If you continue to use this site we will assume that you are happy with it.Ok
