Are CNNs, MLPs, and Classical Neural Networks Obsolete?

A Scientific and Systems-Level Perspective on Modern Artificial Intelligence

Artificial intelligence evolves at a remarkable pace, and each new breakthrough tends to generate the same recurring narrative: that whatever came before is now obsolete. In recent years, the rise of Transformer-based architectures, Large Language Models, and multimodal foundation models has led many to question the relevance of classical neural networks such as Multi-Layer Perceptrons (MLPs) and Convolutional Neural Networks (CNNs). This perception, however, overlooks the nuanced way in which progress in machine learning actually occurs.

Classical architectures have not been replaced; rather, they have become specialized, embedded, and indispensable components of modern AI systems. Understanding their continuing relevance requires moving beyond benchmark hype and examining neural networks from scientific, architectural, and systems-engineering perspectives. The question is therefore not whether CNNs or MLPs are obsolete, but whether they are being evaluated in the correct context.

Multi-Layer Perceptrons (MLPs) represent the most basic form of neural computation. Mathematically, an MLP is a sequence of affine transformations followed by nonlinear activation functions, capable of approximating arbitrary continuous functions under mild conditions. This theoretical property, formalized by the Universal Approximation Theorem, establishes MLPs as powerful function approximators. However, expressive power alone does not guarantee efficiency or generalization. MLPs make almost no assumptions about the structure of the data, which is both their greatest strength and their primary limitation.

In domains where data is structured, low-dimensional, and already encoded in meaningful features, such as finance, healthcare records, industrial telemetry, or business analytics, this lack of inductive bias becomes an advantage. MLPs often outperform more complex architectures in these settings because they introduce less variance, require fewer samples to generalize, and are computationally efficient. For this reason, MLPs remain widely used in production systems, particularly for tabular data. Far from disappearing, they are also deeply embedded within modern architectures. Every Transformer block contains large feed-forward networks that are, in essence, MLPs. Classification heads, regression layers, reinforcement learning policies, and Mixture-of-Experts (MoE) components all rely on MLPs as fundamental building blocks. Declaring MLPs obsolete would therefore imply declaring most modern AI architectures obsolete as well.

Convolutional Neural Networks (CNNs) were introduced to address a different limitation of early neural models: the inability to exploit spatial structure. Images, videos, and other grid-like data exhibit strong local correlations and translational regularities. CNNs encode these assumptions directly through local receptive fields, weight sharing, and hierarchical feature extraction. This architectural bias allows CNNs to learn visual representations efficiently, using far fewer parameters than fully connected networks. From a biological and computational perspective, CNNs are remarkably well aligned with human visual processing. Early layers detect edges and simple patterns, intermediate layers capture textures and shapes, and deeper layers represent object-level semantics. This hierarchical organization enables CNNs to generalize well even when training data is limited. As a result, CNNs remain the backbone of countless real-world systems, including autonomous driving perception stacks, medical imaging diagnostics, satellite image analysis, industrial inspection pipelines, and mobile vision applications.

The emergence of Vision Transformers has led to renewed debate about the future of CNNs. Vision Transformers replace convolutional inductive bias with global self-attention, allowing every image patch to interact with every other patch. While this approach achieves impressive results on massive datasets, it comes at the cost of increased computational complexity and reduced data efficiency. In practice, CNNs often outperform Vision Transformers in small- and medium-scale regimes, especially when latency, energy consumption, and robustness matter. Many state-of-the-art vision systems now adopt hybrid designs, using CNNs for efficient feature extraction and Transformers for high-level reasoning. CNNs are not obsolete; they are efficient specialists integrated into larger systems.

Sequential data introduces yet another structural challenge. Language, speech, sensor streams, and time series depend on temporal order. Recurrent Neural Networks (RNNs) were designed to model this dependency by maintaining a hidden state that evolves over time. Vanilla RNNs, while conceptually elegant, suffer from optimization difficulties that limit their ability to capture long-range dependencies. Long Short-Term Memory (LSTM) networks and Gated Recurrent Units (GRUs) addressed these issues by introducing gating mechanisms that regulate information flow, enabling stable learning over long sequences. Although Transformers have largely replaced recurrent models in large-scale natural language processing, LSTMs and GRUs remain highly relevant in domains where streaming inference, low latency, or limited data are critical. In many real-time systems, such as speech recognition on edge devices or industrial time-series monitoring, recurrent models offer a better trade-off between performance and efficiency than attention-based architectures.

Transformers themselves represent a powerful but often misunderstood paradigm. By relying on self-attention, Transformers can model global dependencies without recurrence, enabling massive parallelization and scalability. This makes them ideal for large datasets and general-purpose modeling, which is why they form the foundation of modern language and multimodal models. Beyond Transformers, the modern AI landscape has grown to include several specialized architectures and training paradigms. Large Language Models (LLMs) are Transformer-based architectures trained on massive text corpora, often extended to multimodal capabilities. Mixture-of-Experts (MoE) architectures distribute computation across specialized subnetworks, improving scalability while retaining high performance. Vision-Language Models (VLMs) integrate visual and textual inputs, combining CNNs or Vision Transformers with MLPs and attention layers to achieve multimodal reasoning. Sequence Learning Models (SLMs) continue to process temporal data efficiently, building on recurrent or Transformer-based foundations, while Masked Language Models (MLMs) employ objectives in which certain tokens are hidden and predicted, as exemplified by BERT. Segment Anything Models (SAMs) demonstrate universal image segmentation capabilities using hybrid CNN and Transformer backbones. Even less standardized terms, such as Latent Concept Models (LCMs) or Language-Aware Models (LAMs), highlight ongoing efforts to adapt classical components to novel data representations and task-specific biases.

Beyond grids and sequences lies relational data, which is best modeled as graphs. Graph Neural Networks (GNNs) introduce a message-passing framework that allows node representations to evolve based on local neighborhood structure. This inductive bias is irreplaceable for tasks involving social networks, molecular structures, recommendation systems, and knowledge graphs. No amount of attention or convolution can fully substitute for architectures explicitly designed to operate on graphs. GNNs are therefore not alternatives to CNNs or Transformers, but complementary tools tailored to relational reasoning.

Generative modeling further illustrates the cumulative nature of AI progress. Autoencoders (AEs) and Variational Autoencoders (VAEs) introduced principled approaches to representation learning and probabilistic modeling. While newer techniques such as diffusion models have achieved superior generative quality, they rely heavily on classical components. Most diffusion models use convolutional U-Net architectures augmented with attention layers, reinforcing the idea that innovation builds on existing foundations rather than discarding them.

At the systems level, modern AI increasingly relies on modularity. Mixture-of-Experts architectures distribute computation across specialized subnetworks, improving scalability and efficiency. These experts are not novel primitives but combinations of Transformers, CNNs, and MLPs. Even cutting-edge Large Language Models depend on classical neural components at their core.

From a scientific perspective, the concept of architectural obsolescence is misguided. Neural networks encode assumptions about the structure of data. When those assumptions align with the problem domain, performance and efficiency follow. When they do not, even the most advanced model will struggle. Progress in artificial intelligence is therefore best understood as specialization and integration, not replacement. Classical networks, modern architectures, and contemporary paradigms such as LLMs, MoEs, VLMs, SLMs, MLMs, and SAMs each occupy a distinct and essential niche. Modern AI systems succeed not because one architecture dominates, but because multiple architectures are combined intelligently.

LLM – Large Language Model

What it is:

• A neural network, usually Transformer-based, trained on massive text corpora.
  What it does:

• Generates and understands text, answers questions, translates languages, summarizes, and can even reason over long documents.
  Relation to classical models:

• Uses MLPs in feed-forward layers and attention mechanisms. Transformers are built on classical building blocks.

LCM – Latent Concept Model

What it is:

• A model designed to extract and represent hidden (latent) concepts from data, often in an interpretable or structured latent space.
  What it does:

• Learns abstract features or concepts that may not be directly observable, useful in tasks like recommendation systems or multimodal reasoning.
  Relation to classical models:

• Often uses MLPs, autoencoders, or CNNs to extract latent features.

LAM – Language-Aware Model

What it is:

• A model that incorporates language understanding into other tasks, e.g., reasoning over text plus another modality.
  What it does:

• Enhances models in vision, graphs, or multimodal tasks by integrating textual context.
  Relation to classical models:

• Combines MLPs, CNNs, Transformers, depending on the primary input type.

MoE – Mixture-of-Experts

What it is:

• A modular architecture with multiple “expert” subnetworks, where only a subset is active for each input.
  What it does:

• Improves scalability and efficiency, allowing very large models without fully computing every parameter for each input.
  Relation to classical models:

• Each “expert” can be an MLP, CNN, or Transformer block—MoE is a system-level design, not a new primitive.

VLM – Vision-Language Model

What it is:

• A model that combines visual input (images/video) with textual input.
  What it does:

• Can answer questions about images, generate captions, or reason across modalities.
  Relation to classical models:

• Uses CNNs or Vision Transformers for images, MLPs in classification heads, and attention layers for multimodal reasoning.

SLM – Sequence Learning Model

What it is:

• A model designed specifically for temporal or sequential data (time series, language, sensor streams).
  What it does:

• Learns patterns over sequences for prediction, anomaly detection, or control tasks.
  Relation to classical models:

• May use RNNs, LSTMs, GRUs, or Transformers depending on scale and latency requirements.

MLM – Masked Language Model

What it is:

• A model trained to predict missing tokens in text, e.g., BERT.
  What it does:

• Learns contextual embeddings and deep language understanding.
  Relation to classical models:

• Transformer-based, with MLPs in feed-forward layers for token prediction.

SAM – Segment Anything Model

What it is:

• A universal image segmentation model that can identify objects in images with minimal guidance.
  What it does:

• Can segment any object in an image, supports zero-shot segmentation for unseen classes.
  Relation to classical models:

• Typically uses CNN backbones for feature extraction combined with Transformers for reasoning and mask prediction.

Training Techniques in PyTorch: Early Stopping, Dropout, and Regularization

For our example, we use the Fashion-MNIST dataset, although the specific dataset is not particularly important for demonstrating these concepts.

Initial Setup

We begin by importing the required libraries and loading our dataset. The model is configured to train for eight epochs, which represents the maximum number of epochs the training process is allowed to run.

However, thanks to early stopping, the training may end before reaching this limit.

Implementing Early Stopping in PyTorch

Since PyTorch does not provide built-in early stopping functionality, we must implement it manually within the training loop.

Tracking the Best Validation Loss

Early stopping is based on monitoring the validation loss. We start by initializing the best validation loss to infinity. This guarantees that the first computed validation loss will always be an improvement.

Defining an Improvement Threshold

Not every improvement is meaningful. To avoid reacting to very small fluctuations, we define a minimum improvement threshold equal to 0.001. If the validation loss fails to improve by at least this value, the epoch is considered below the performance threshold.

Setting Patience

The patience parameter determines how many times the validation loss is allowed to fall below the threshold before training is stopped. In this example, patience is set to two, meaning that training will stop early if the validation loss does not sufficiently improve two times.

Early Stopping Logic in the Training Loop

At the end of each epoch, we compute the validation loss and calculate the difference between the best validation loss and the current validation loss. If the current validation loss is lower, we update the best value. If the improvement does not meet the threshold, a counter is increased. When this counter reaches the patience value, the training loop is interrupted.

This logic allows the model to stop training once it stops learning meaningful patterns.

Dropout for Regularization

Another important technique covered in this demo is dropout, which helps prevent overfitting. In PyTorch, dropout is implemented as a layer, where the parameter p represents the probability that a neuron is zeroed out.

With p set to 0.5, half of the inputs to the dropout layer are randomly set to zero during training. One advantage of PyTorch’s implementation is that the same dropout layer can be reused throughout the model without worrying about input or output sizes.

Regularization and Momentum in the Optimizer

Both L2 regularization and momentum are implemented directly in the optimizer. Regularization is controlled using the weight decay parameter, while momentum is controlled using the momentum parameter.

When using stochastic gradient descent, these options help stabilize training and improve the model’s ability to generalize.

Putting Everything Together

All of these techniques—early stopping, dropout, regularization, and momentum—can be used simultaneously in a single training pipeline.

After training the model, we observe that all eight epochs are completed, even though early stopping is enabled. This happens because the validation loss only fails to improve beyond the threshold in Epoch 6 and Epoch 8. Since the patience value is two, training is allowed to continue.

If the model had been configured to train for ten epochs instead of eight, early stopping would have triggered at Epoch 8

How to Write a Training Loop in PyTorch: A Practical Guide

Getting Started: Model, Data, and Training Function

To keep our code clean and reusable, we wrap the training loop inside a function called train_model.
This function takes the following parameters:

• the model

• the number of epochs (with a default value of 8)

• the learning rate (default set to 0.05)

Inside this function, we can either define the optimizer and loss function directly or pass them in as parameters. In this example, we use:

• Stochastic Gradient Descent (SGD) with momentum = 0.9

• Cross Entropy Loss, implemented with nn.CrossEntropyLoss

This setup is commonly used for classification tasks.

The Core of the Training Loop

Now let’s dive into the most important part: the training loop.

For each epoch:

1. Set the model to training mode using model.train()
   This ensures that gradients are computed correctly and that layers such as Dropout and Batch Normalization behave as expected during training.

2. Initialize tracking variables

   • epoch loss

   • number of correct predictions

3. Iterate over the training data
   We use enumerate(train_loader) to keep track of the number of mini-batches processed. This is especially useful when working with large datasets or long training times, as it allows us to monitor running loss during each epoch.

4. GPU handling
   We check whether a GPU is available with torch.cuda.is_available().
   If so, we move both inputs and labels to the GPU using the .cuda() method.

5. Forward pass

   • reset gradients with optimizer.zero_grad()

   • pass the inputs through the model to obtain outputs

6. Loss computation
   We compute the loss by passing the model outputs and the labels to the loss function (nn.CrossEntropyLoss).

7. Backward pass and parameter update

   • compute gradients using loss.backward()

   • update model parameters with optimizer.step()

8. Accuracy calculation
   We obtain predictions using torch.max on the output tensor and compare them with the ground-truth labels to count the number of correct predictions.

Model Validation

After completing the training phase for an epoch, we move on to validation:

• initialize validation loss and correct prediction counters

• set the model to evaluation mode with model.eval()
  This disables gradient computation and ensures consistent behavior during inference

• perform a forward pass on the validation dataset

• compute validation loss and accuracy in the same way as during training

Finally, we print the training and validation metrics to the notebook.

Running the Training Process

At this point, all that’s left to do is call the train_model function and pass in the instantiated model (for example, net).
Depending on the dataset size and model complexity, training may take some time.

 

Machine Learning Training Concepts Help

Training data is the dataset actually used to fit the model.

The model learns patterns, relationships, and parameters directly from this data.

Example

from sklearn.model_selection import train_test_split

X_train, X_temp, y_train, y_temp = train_test_split(
    X, y, test_size=0.3, random_state=42
)

Validation Data

Validation data is used during training to evaluate how well the model generalizes to unseen data.
It helps tune hyperparameters and detect overfitting.

Example

X_val, X_test, y_val, y_test = train_test_split(
    X_temp, y_temp, test_size=0.5, random_state=42
)

Test Data

Test data is used only once, after training is complete, to evaluate final model performance.

Example

from sklearn.metrics import accuracy_score

y_pred = model.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
print("Test accuracy:", accuracy)

Underfitting (Bias Error)

Underfitting happens when the model is too simple to capture the underlying structure of the data.
The model performs poorly on both training and validation sets.

Example

from sklearn.linear_model import LinearRegression

model = LinearRegression()
model.fit(X_train, y_train)

print("Training score:", model.score(X_train, y_train))
print("Validation score:", model.score(X_val, y_val))

Overfitting (Variance Error)

Overfitting occurs when the model is too complex and fits the training data too closely.
It performs well on training data but poorly on new data.

Example

from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline

model = make_pipeline(
    PolynomialFeatures(degree=10),
    LinearRegression()
)

model.fit(X_train, y_train)
print("Training score:", model.score(X_train, y_train))
print("Validation score:", model.score(X_val, y_val))

Early Stopping

Early stopping halts training when validation error stops improving, preventing overfitting.

Example

from tensorflow.keras.callbacks import EarlyStopping

early_stop = EarlyStopping(
    monitor="val_loss",
    patience=5,
    restore_best_weights=True
)

model.fit(
    X_train, y_train,
    validation_data=(X_val, y_val),
    epochs=100,
    callbacks=[early_stop]
)

Dropout

Dropout randomly disables neurons during training, forcing the network to learn more robust features.

Example

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout

model = Sequential([
    Dense(128, activation="relu"),
    Dropout(0.5),
    Dense(64, activation="relu"),
    Dropout(0.5),
    Dense(1)
])

Local Minima

A local minimum is a point where the loss is minimal in a small region, but not globally optimal.
Gradient-based optimizers can get stuck in these points.

Example

import numpy as np

def loss(x):
    return x**4 - 3*x**2 + 2

x = np.linspace(-3, 3, 100)
y = loss(x)

Momentum

Momentum improves gradient descent by adding a fraction (β between 0 and 1) of the previous update to the current one.

Example

from tensorflow.keras.optimizers import SGD

optimizer = SGD(
    learning_rate=0.01,
    momentum=0.9
)

model.compile(
    optimizer=optimizer,
    loss="mse"
)

Stochastic Gradient Descent (SGD)

Stochastic Gradient Descent updates model parameters using small subsets of the data (mini-batches), making training faster and noisier—but often more effective.

Example

model.compile(
    optimizer="sgd",
    loss="categorical_crossentropy",
    metrics=["accuracy"]
)

model.fit(
    X_train, y_train,
    batch_size=32,
    epochs=50
)

How to Use TensorBoard with PyTorch

TensorBoard allows you to visualize:

• Training and validation loss

• Accuracy and other custom metrics

• Images, model graphs, and embeddings

• Comparisons between multiple training runs

Instead of relying only on printed logs, TensorBoard provides an interactive dashboard that makes it much easier to track what is happening during training.

Installing TensorBoard

If you are using PyTorch, TensorBoard is usually installed automatically. If not, you can install it manually:

pip install tensorboard

Make sure you also have PyTorch installed:

pip install torch torchvision

Launching TensorBoard

TensorBoard is started from the command line. First, choose a directory where your training logs will be stored (for example, runs/). Then run:

tensorboard --logdir=runs

After launching, TensorBoard will print a local URL, typically:

http://localhost:6006

Open this URL in your browser to access the TensorBoard dashboard.

Using SummaryWriter in PyTorch

PyTorch provides TensorBoard integration through the SummaryWriter class, which is located in:

from torch.utils.tensorboard import SummaryWriter

The SummaryWriter is responsible for writing event files that TensorBoard reads and visualizes.

Creating a SummaryWriter

writer = SummaryWriter(log_dir="runs/experiment_1")

Each experiment should ideally have its own log directory. This makes it easy to compare multiple runs in TensorBoard.

Logging Scalars (Loss, Accuracy, Metrics)

The most common use of TensorBoard is logging scalar values such as loss and accuracy during training.

Example: Logging Training Loss

for epoch in range(num_epochs):
    train_loss = 0.0
    
    # training loop
    for inputs, labels in dataloader:
        outputs = model(inputs)
        loss = criterion(outputs, labels)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        train_loss += loss.item()

    avg_loss = train_loss / len(dataloader)
    writer.add_scalar("Loss/train", avg_loss, epoch)

Logging Accuracy

writer.add_scalar("Accuracy/train", accuracy, epoch)

In TensorBoard, these values will appear as smooth, interactive plots that update as training progresses.

Logging Images

TensorBoard can also display images, which is especially useful for computer vision tasks.

Example: Logging a Batch of Images

from torchvision.utils import make_grid

images, labels = next(iter(dataloader))
img_grid = make_grid(images)

writer.add_image("Training Images", img_grid)

You can view these images in the Images tab of TensorBoard.

Logging Figures

Sometimes you may want to log custom plots, such as confusion matrices or matplotlib charts. TensorBoard supports this using the add_figure method.

Example: Logging a Matplotlib Figure

import matplotlib.pyplot as plt

fig, ax = plt.subplots()
ax.plot([1, 2, 3], [4, 5, 6])
ax.set_title("Sample Plot")

writer.add_figure("My Figure", fig, global_step=epoch)

This is useful for visualizing advanced statistics or custom evaluation results.

Closing the Writer

After training is complete, always close the SummaryWriter to ensure all data is written to disk:

writer.close()

Best Practices

• Use clear tag names (e.g., Loss/train, Loss/val)

• Create separate log directories for different experiments

• Log metrics at consistent intervals

• Avoid logging too frequently to reduce disk usage

Understanding Neural Network Decision Boundaries

When it comes to data classification, one of the key concepts is the decision boundary: how an algorithm separates different classes in a dataset. In this article, we’ll explore what these boundaries look like, from simple lines to the complex hyperplanes of neural networks.

Linear Decision Boundaries

The simplest type of decision boundary is linear. Imagine a 2D dataset with two input variables. In this case, we can separate two classes using a straight line that divides the points.

This approach works well for binary classes in two dimensions. However, in practice, we rarely use a neural network for such simple problems: linear boundaries are easily handled with simpler methods like logistic regression or decision trees.

From Lines to Hyperplanes

As we increase the number of dimensions (the number of input variables), the concept of a linear decision boundary extends:

• In 3D, the boundary becomes a plane.

• In higher dimensions, it becomes a hyperplane, a concept that is hard to visualize but mathematically straightforward.

For a dataset with n variables, a linear decision boundary will be an (n-1)-dimensional hyperplane. Even though we can’t easily visualize beyond 3D, we can think of it as an extension of a plane that divides space.

The limitation of linear boundaries is that they cannot handle complex or non-linear data. When classes are not separable by a line or plane, more advanced methods are required.

Neural Network Decision Boundaries

Neural networks shine in cases where the data is non-linear and high-dimensional. Each neuron introduces a small hyperplane, and by combining these hyperplanes across the network, we get complex decision boundaries, often in hundreds of dimensions.

These boundaries are no longer linear, allowing the network to fit very intricate datasets. However, their high dimensionality makes them nearly impossible to visualize directly.

Visualizing Decision Boundaries

To better understand how neural networks separate classes, we can use dimensionality reduction techniques like t-SNE (t-Distributed Stochastic Neighbor Embedding).
t-SNE allows us to project high-dimensional data into 2D or 3D while preserving the class structure, giving us an approximate idea of the network’s decision boundaries.

When Large Neural Networks Are Needed

If the dataset is particularly complex, the required decision boundary will also be complex. In such cases, we need a larger neural network, with more neurons and layers, capable of modeling the intricate structure of the data.

In summary:

• Linear boundaries are simple and interpretable but limited.

• Neural networks create non-linear, high-dimensional boundaries, ideal for complex data.

• Visualization is challenging but can be approximated using techniques like t-SNE.

Understanding decision boundaries is crucial for designing effective neural networks and choosing the right architecture for the problem you want to solve.

How to Choose Output Activation Functions in Neural Networks

Neural networks are powerful tools for tackling prediction and classification problems. However, one of the most important decisions when building a network is choosing the output activation function. This choice directly affects the shape and range of your model’s predictions, impacting overall performance.

In this article, we’ll guide you on how to select the right activation based on your problem type: binary classification, multi-class classification, or regression.

What is an Output Activation Function?

The output activation function transforms the values produced by the last layer of your network into a format that can be interpreted as predictions.

Specifically, it determines:

• The shape of the output – a single number, a vector, or a tensor.

• The range of the output – a continuous number or a probability between 0 and 1.

It is essential that your network’s output matches the shape of your labels.

Problem Types and Recommended Activations

1. Binary Classification

Binary classification is when your task is to choose between two possibilities (e.g., “Is this a cat or not?”).

• The network predicts the probability that an example belongs to the positive class.

• Recommended activation: sigmoid, which maps any input to a value between 0 and 1.

• Loss function to minimize: binary cross-entropy.

Example: If you want to predict whether an image contains a cat, the output might be 0.87, meaning an 87% chance that it’s a cat.

2. Multi-Class Classification

When there are more than two classes (e.g., cat, dog, fish, rabbit), it’s a multi-class classification problem.

• The output is a vector of probabilities, one for each class.

• Recommended activation: softmax, which converts scores into a probability distribution that sums to 1.

• Loss function: multi-class cross-entropy.

• The predicted class is the one with the highest probability, often determined with the argmax function.

Example: For an image of a cat, the output might be [0.9, 0.05, 0.03, 0.02], indicating a 90% probability for the “cat” class.

3. Regression

Regression problems involve predicting a continuous numerical value (e.g., the price of a house).

• The output is the predicted value itself.

• Recommended activation: identity (no activation) or ReLU if you want to ensure positive outputs.

• Common loss functions: mean squared error (MSE) or mean absolute error (MAE).

Example: Predicting the price of a house might yield an output of 250000.

Key Takeaways

The choice of output activation function is never arbitrary; it depends on your problem type and the shape of your labels:

• Sigmoid → binary classification

• Softmax → multi-class classification

• Identity/ReLU → regression Mean Squared Error 

 

Focus on Activation functions

Activation functions are a fascinating area of research in machine learning. They are a special class of mathematical functions characterized by several important properties. First of all, activation functions are usually non-linear. If all activation functions were linear, the entire neural network would collapse into a simple linear model, losing its expressive power.

Another key requirement is differentiability. Ideally, an activation function should be continuously differentiable, meaning it has a derivative everywhere. There is, however, one very popular activation function that is differentiable almost everywhere rather than everywhere.

Activation functions should also be monotonic, meaning that as the input increases, the output never decreases. In other words, when moving from left to right along the x-axis, the value of the function does not go down. Many commonly used activation functions satisfy this property.

A further important characteristic is that the activation function should approximate the identity function near the origin. This ensures that, around zero, the input is passed through without significant distortion.

Let us now look at some of the most common activation functions.

The most widely used activation function is the ReLU (Rectified Linear Unit). In many neural networks, ReLU is the default choice for hidden layers because it performs very well in practice and makes gradient computation efficient. However, ReLU lacks one of the desirable properties mentioned earlier: it is not differentiable at zero.

To address this limitation, the Leaky ReLU was introduced. Instead of outputting zero for negative inputs, Leaky ReLU outputs a small fraction of the input, allowing gradients to flow even for negative values.

In older research papers, the sigmoid activation function is frequently encountered. Sigmoid has some appealing properties: it smoothly maps inputs to values between 0 and 1 and has an easily computable derivative. However, in practice, neural networks trained with sigmoid tend to perform worse than those using ReLU, which is why sigmoid is rarely used in hidden layers today.

Another commonly used activation function is the hyperbolic tangent (tanh). Like sigmoid, tanh is bounded, but its outputs range from −1 to 1. One of its advantages is that it is centered at zero and has larger derivatives than sigmoid, making it a better choice in many cases.

There is also the step function, which is encountered in the perceptron model. While simple, it is not suitable for gradient-based learning.

Finally, in more advanced research, more esoteric activation functions may be encountered, such as the Gaussian Error Linear Unit (GELU). GELU has been adopted in several transformer-based models, including OpenAI’s GPT-3.

How to Add a Table of Contents to Your Jupyter Notebook

Working inside Jupyter notebooks is one of the most popular ways to write and share code, especially in data science, machine learning, and research workflows. As notebooks grow in length and complexity, navigating them can become frustrating. This is where a Table of Contents becomes extremely useful.

In this article, we’ll look at why a Table of Contents matters, how to create one manually, how anchors work in Jupyter, and how modern tools can generate a TOC automatically.

Why Your Notebook Needs a Table of Contents

Long notebooks with multiple sections such as data loading, preprocessing, analysis, and visualization are hard to navigate by scrolling alone. A Table of Contents helps by:

• Improving navigation

• Making the structure of the notebook clear

• Enhancing readability for collaborators

• Providing a quick overview of the workflow

This is particularly helpful when sharing notebooks or exporting them to HTML or PDF.

Creating a Manual Table of Contents with Markdown

One simple way to create a Table of Contents is by using Markdown links that point to section headers.

Example:

### Table of Contents

1. [Introduction](#introduction)
2. [Load Data](#load-data)
3. [Exploratory Analysis](#exploratory-data-analysis)
4. [Results](#results)
5. [Conclusion](#conclusion)

Each link refers to a specific section inside the notebook.

How Section Anchors Work in Jupyter Notebooks

In Jupyter notebooks, you usually do not need to manually define anchors using HTML. When you create a Markdown heading, Jupyter automatically generates an anchor for it.

Example:

## Introduction

Jupyter automatically creates the anchor:

#introduction

This means you can link to that section using:

[Introduction](#introduction)

Automatic Anchor Naming Rules

Jupyter follows these rules when creating anchors from headings:

• All characters are converted to lowercase

• Spaces are replaced with hyphens

• Special characters are removed

Example:

## Exploratory Data Analysis

Generated anchor:

#exploratory-data-analysis

Link reference:

[Exploratory Analysis](#exploratory-data-analysis)

When to Use <a id=""></a> Manually

In most cases, Markdown headings are enough. However, manually defining an anchor using HTML can be useful in advanced scenarios, such as:

• When you want a custom anchor name

• When linking to a specific point that is not a heading

• When you have duplicate section titles

Example:

<a id="custom-anchor"></a>
### My Section Title

You can then reference it like this:

[Go to section](#custom-anchor)

Automatic Table of Contents Options

If you prefer automation, there are powerful tools available.

Jupyter Notebook Extensions

The Table of Contents (2) extension from jupyter_contrib_nbextensions automatically generates a navigation panel based on notebook headings. It supports automatic updates, collapsible sections, and sidebar display.

JupyterLab Built-in Table of Contents

JupyterLab includes a built-in Table of Contents panel in the sidebar. It reads Markdown headers and allows quick navigation without installing additional extensions.

Real Video Streaming: Why Base64 Fails and How to Do It Properly

If you’ve ever been told, “You can just send video in Base64 inside JSON,” stop right there. Base64 works for small images, debugging, or prototypes, but for serious video—especially live or large files—it’s a disaster: +33% bandwidth, extra CPU, higher latency, wasted memory. No exceptions.

Here’s how real streaming works, with raw bytes, chunking, and codecs, plus a practical WebSocket example.

Why Base64 Doesn’t Work

Base64 introduces serious overhead:

• Bandwidth: +33% compared to raw binary

• CPU: Encoding/decoding is expensive

• Latency: Every chunk must be converted

• Memory: Extra buffers for the string

• Scalability: Impossible for live or large files

Base64 is only useful for tiny assets or debugging, never production.

How Real Streaming Works

1. Raw Binary Stream

Send the video as pure bytes, never text. Options include:

• TCP / UDP

• WebSocket (binary)

• HTTP chunked transfer

• QUIC

2. Chunking

Break the video into blocks:

[chunk][chunk][chunk]

• Typical size: 1–64 KB

• Sequential order

• No text encoding

3. Codec

You don’t send raw frames; codecs compress video efficiently:

• H.264 / AVC (standard)

• H.265 / HEVC (high efficiency)

• VP9 / AV1 (open source, high quality per bandwidth)

The server sends compressed bytes, the client decodes them in real time.

Conceptual Example

Server (encoder):

Camera → H.264 encoder → byte stream → socket

Client (decoder):

socket → byte stream → decoder → frame → display

Web Options

WebRTC	Live, low latency
HLS	On-demand streaming
DASH	Adaptive streaming
WebSocket (binary)	Custom real-time prototypes

Minimal WebSocket Example (JS)

Server Node.js:

videoChunks.forEach(chunk => {
  ws.send(chunk); // Pure binary, no strings
});

Client Browser:

const socket = new WebSocket("wss://example.com/video");
socket.binaryType = "arraybuffer";

socket.onmessage = (event) => {
  const chunk = new Uint8Array(event.data);
  // Decode using MediaSource or WebCodecs
};

Notice: no Base64, only raw bytes.

Real Pipeline with FFmpeg (Conceptual)

ffmpeg -i input.mp4 \
       -c:v libx264 -preset ultrafast -f mpegts udp://127.0.0.1:1234

• libx264 → codec

• -f mpegts → streaming-friendly container

• udp:// → raw stream transmission

The client receives packets and feeds them directly into the decoder.

Base64 is fine for prototypes. For large files or live streaming, you need pure binary, chunking, codecs, and a proper protocol. With WebRTC, HLS, DASH, or binary WebSocket, you get low latency, high quality, and real scalability.

In a follow-up, I can show a full demo:

• Convert a photo/video into a raw byte stream

• Simulate frame → chunk → stream

• Build a real WebRTC + FFmpeg production-ready pipeline

This is the serious world of video streaming. 

Why Base64 Is Killing Your App’s Performance (And What to Use Instead)

The Hidden Cost of Base64

Base64 was never designed for data storage or high-volume file transfers.
Its original purpose was to move binary data through systems that only understood plain text.

When you use it for large payloads today, you end up paying three major taxes.

1. The 33% Size Tax

Base64 encodes every 3 bytes of binary data as 4 ASCII characters.
The result is a ~33% increase in size, every single time.

A 100 MB video suddenly becomes 133 MB of text:

• more bandwidth consumed

• more time spent uploading

• more storage wasted

All for zero functional benefit.

2. The Memory Bottleneck

Most Base64 encoders and decoders require the entire payload to be loaded into memory.

That means a 500 MB upload can easily cause:

• 1 GB RAM spikes

• garbage collection pressure

• process crashes under load

One large request can bring an otherwise healthy server to its knees.

3. The CPU Overhead

Encoding and decoding Base64 is not free.

Your CPU must:

• parse large strings

• convert them back to binary

• allocate new buffers

All of this adds latency, increases response times, and reduces overall throughput—especially under concurrent load.

3 Better Alternatives for Large Files

If you’re building a scalable, production-grade system, stop treating files as strings.
Modern architectures are binary-first.

Here are three proven approaches.

1. The Cloud-Direct Pattern (Presigned URLs)

Your API should not be a middleman for raw bytes.

Instead of:

Client → API → Object Storage

Use presigned URLs.

How it works
The client asks your API for permission.
Your API returns a short-lived, secure upload URL from providers like AWS S3 or Google Cloud Storage.
The client uploads the file directly to the cloud.

Why it wins

• Zero file data touches your server

• No memory spikes

• No CPU overhead

• Massive scalability for free

Your backend stays fast and boring. Exactly how it should be.

II. Chunked & Resumable Uploads (TUS Protocol)

Uploading a 2 GB file in a single request is a gamble.

If the connection drops at 99%, the user starts over—and hates you for it.

How it works
Split the file into small chunks (e.g. 5 MB).
Upload them sequentially using a resumable protocol like TUS.

Why it wins

• Fault-tolerant by design

• Uploads can resume after failures

• Ideal for unstable networks and large files

This is the standard for serious upload workflows.3. Binary Streaming

Sometimes you do need the file on your server—virus scanning, media processing, transformations.

In that case, stream it.

How it works
Use multipart/form-data and process the incoming request as a stream.
Pipe the data chunk-by-chunk directly to disk, cloud storage, or a processing pipeline.

Why it wins

• Constant, predictable memory usage

• Works with arbitrarily large files

• Plays nicely with backpressure

Streams are how servers are meant to handle data.

Base64 is fine for:

• small icons

• tiny blobs

• email attachments

But for large files, it’s pure technical debt.

If you want faster uploads, lower costs, and servers that don’t fall over when someone uploads a 4K video, move to presigned URLs, chunked uploads, or streaming.

Extracting Tables from PDFs in Python: A Practical Comparison of Tools

Working with PDFs is one of the most common (and frustrating) data engineering tasks. Unlike CSV or Excel files, PDFs are designed for visual presentation, not structured data extraction. Choosing the right Python library can save you hours of cleanup or completely break your pipeline.

In this article, we compare the most popular Python tools for extracting tables and text from PDFs, focusing on accuracy, complexity, and real-world use cases.

Quick Comparison Overview

Different tools shine in different scenarios. Here is a high-level summary before diving deeper:

Tabula-py is best for clean, well-structured tables.
Camelot is excellent for wide and complex layouts.
pdfplumber is flexible and powerful for irregular tables.
PyMuPDF is fast for text extraction but needs extra parsing.
Tesseract OCR is the only option for scanned PDFs.
pdfquery is perfect when exact coordinates are required.

Tabula-py

Type: Text-based
Output: Pandas DataFrame
Complexity: Medium

Tabula-py is a Python wrapper for Tabula (Java-based) and is one of the most popular tools for table extraction.

Pros:

• Very easy to use

• Direct output as Pandas DataFrames

• Great results on clean, grid-based tables

Cons:

• Struggles with complex layouts

• Requires Java

Best use case: clean, well-formatted tables with clear borders.

Camelot

Type: Text-based
Output: Pandas DataFrame
Complexity: Medium

Camelot is often considered more accurate than Tabula, especially for wide tables or complex page layouts.

Pros:

• Excellent precision

• Handles complex table structures better than Tabula

• Supports both lattice and stream parsing modes

Cons:

• Slightly steeper learning curve

• Can fail on very irregular tables

Best use case: wide tables and complex layouts where precision matters.

pdfplumber

Type: Text-based with parsing
Output: Requires processing
Complexity: Medium–High

pdfplumber offers low-level access to PDF elements and is extremely flexible.

Pros:

• Highly customizable

• Excellent for irregular or borderless tables

• Can extract text, lines, and coordinates

Cons:

• Requires manual parsing logic

• More coding effort compared to Tabula or Camelot

Best use case: irregular tables or PDFs where automated tools fail.

PyMuPDF (fitz)

Type: Text-based
Output: Text only
Complexity: Medium

PyMuPDF is fast and efficient but does not natively extract tables.

Pros:

• Very fast

• High-quality text extraction

• Good for preprocessing PDFs

Cons:

• No built-in table extraction

• Requires custom parsing

Best use case: fast text extraction when you plan to build your own table parser.

Tesseract OCR

Type: Image-based
Output: Text only
Complexity: High

When PDFs are scanned images, OCR is the only viable solution.

Pros:

• Works with scanned PDFs

• Supports multiple languages

Cons:

• Lower accuracy than text-based tools

• No table awareness

• Requires image preprocessing

Best use case: scanned documents with no embedded text.

pdfquery

Type: Text-based
Output: Text with coordinates
Complexity: High

pdfquery is ideal when you need pixel-level control.

Pros:

• Precise coordinate-based extraction

• Ideal for fixed-layout documents

• Powerful for automation

Cons:

• Complex setup

• Not beginner-friendly

Best use case: PDFs with consistent layouts where exact positioning matters. 

Detecting Dynamics 365 Web Resource Updates with JavaScript Hashing

When working with Dynamics 365 / Dataverse Web Resources, one common challenge is ensuring users are aware when a JavaScript or HTML Web Resource has been updated. Browser caching can cause users to unknowingly run outdated versions, leading to unexpected behavior and hard-to-diagnose bugs.

In this article, we’ll look at a simple and effective technique to detect Web Resource changes at runtime using JavaScript hashing and notify users when an update is detected.

The Idea

The core idea is straightforward:

1. Download the Web Resource content

2. Calculate a hash (SHA-1) of the content

3. Store the hash in localStorage

4. Compare the current hash with the previously stored one

5. Notify the user if the hash has changed

This approach works entirely on the client side and requires no server-side customization.

Triggering the Check on Form Load

The check is executed when a Dynamics 365 form loads:

formContext.data.addOnLoad(
  Opportunity.checkWebResourceHash.bind(this, "opportunity_webresource")
);

This ensures the verification runs automatically whenever the form is opened.

The Hash Comparison Function

Here’s the full implementation of the function responsible for detecting changes:

checkWebResourceHash: async function (webResourceName) {
  const STORAGE_KEY = `WR_HASH_${webResourceName}`;
  const STORAGE_DATE_KEY = `WR_HASH_DATE_${webResourceName}`;

  try {
    const clientUrl = Xrm.Utility.getGlobalContext().getClientUrl();
    const url = `${clientUrl}/WebResources/${webResourceName}`;

    // Fetch the Web Resource content without using cache
    const response = await fetch(url, { cache: "no-store" });
    const text = await response.text();

    // Generate SHA-1 hash
    const encoder = new TextEncoder();
    const data = encoder.encode(text);
    const hashBuffer = await crypto.subtle.digest("SHA-1", data);

    const newHash = Array.from(new Uint8Array(hashBuffer))
      .map(b => b.toString(16).padStart(2, "0"))
      .join("");

    const oldHash = localStorage.getItem(STORAGE_KEY);

    // Detect changes
    if (oldHash && oldHash !== newHash) {
      const now = new Date().toISOString();

      localStorage.setItem(STORAGE_KEY, newHash);
      localStorage.setItem(STORAGE_DATE_KEY, now);

      Xrm.Navigation.openAlertDialog({
        title: "Web Resource Update",
        text:
          `The Web Resource "${webResourceName}" has been updated.\n\n` +
          `Date: ${new Date(now).toLocaleString()}`
      });
    }

    // First-time initialization
    if (!oldHash) {
      localStorage.setItem(STORAGE_KEY, newHash);
      localStorage.setItem(STORAGE_DATE_KEY, new Date().toISOString());
    }

  } catch (e) {
    console.error("Error while checking Web Resource", webResourceName, e);
  }
}

Why SHA-1?

SHA-1 is not recommended for security purposes, but in this scenario it’s perfectly adequate:

• We’re not securing sensitive data

• We only need a fast and consistent checksum

• It’s widely supported by the Web Crypto API

If you prefer, you can easily switch to SHA-256 by replacing "SHA-1" with "SHA-256".

Benefits of This Approach

• No server-side changes

• Works with any JavaScript or HTML Web Resource

• Prevents silent cache-related issues

• Improves transparency for users and testers

• Easy to reuse across multiple forms and entities

Possible Enhancements

• Automatically reload the page after detection

• Display the last update date in a custom notification

• Store hashes per environment (Dev / Test / Prod)

• Extend the logic to multiple Web Resources at once

How to Enable Developer Tools in the New Microsoft Teams Desktop App

If you’re a developer or power user, there may be times when you need access to developer tools in Microsoft Teams. These tools are useful for inspecting elements, debugging custom apps, or troubleshooting layout issues. In the new Microsoft Teams desktop client, however, developer tools are hidden by default. Fortunately, there is a simple workaround to enable them.

Why Developer Tools Are Useful

In the web version of Microsoft Teams, opening developer tools is easy using F12 on Windows or Command + Option + I on macOS. These tools allow you to inspect HTML and CSS, view console logs, and debug issues. In the new desktop client, this functionality is not exposed in the interface, which can be frustrating for developers.

Below is a step-by-step guide to enabling developer tools in the new Teams client.

Step 1 – Create the Configuration File

Start by opening a plain text editor such as Notepad or Visual Studio Code. Create a new file and name it configuration.json. Inside the file, add the following content:

{
  "core/devMenuEnabled": true
}

Save the file to the following location on your Windows machine:

%LOCALAPPDATA%\Packages\MSTeams_8wekyb3d8bbwe\LocalCache\Microsoft\MSTeams

This folder is part of the local cache used by the new Microsoft Teams client.

Step – Fully Close Microsoft Teams

Make sure Microsoft Teams is completely closed. Simply closing the window is not enough. Look for the Teams icon in the system tray, right-click it, and select “Quit”. Once the application is fully closed, reopen Microsoft Teams.

Step – Open Developer Tools

After restarting Teams, right-click the Teams icon in the system tray again. You should now see a new option called “Engineering Tools”. From there, select “Open Dev Tools”.

A separate Developer Tools window will open, allowing you to inspect elements, view logs, and debug just like you would in a web browser.

Additional Notes

You may be asked to sign out and sign back in after enabling developer tools. This behavior is normal. Keep in mind that this feature is not officially documented by Microsoft and may change in future updates, but it is very useful for debugging custom Teams applications and extensions.

Even though the new Microsoft Teams client hides developer tools by default, enabling them is quick and easy with this configuration file trick. Once activated, you gain powerful debugging capabilities that can greatly improve your development and troubleshooting workflow.

Detecting Microsoft Teams Context in Dynamics 365 Using JavaScript

When working with Dynamics 365 or Power Apps embedded inside Microsoft Teams, developers often need to understand where the application is actually running.

Is it opened directly in a browser?
Is it hosted inside Microsoft Teams?
Is it loaded inside an iframe?

This article explains why window.location.href can be misleading in Teams integrations and how to detect the execution context reliably using JavaScript.

The Problem: URL Confusion in Microsoft Teams

When a Dynamics 365 app is opened inside Microsoft Teams, it is wrapped in multiple layers:

• Microsoft Teams shell

• Power Apps web player

• Dynamics 365 Unified Interface (UCI)

• Your page, usually inside an iframe

Because of this architecture, checking the URL with JavaScript like this:

const url = window.location.href;
console.log(url);

can return very different results depending on where the script runs.

Common URL Results You May Encounter

When Dynamics 365 is embedded in Teams, the URL often looks like this:

https://org.crm4.dynamics.com/uclient/main.htm?...&source=teamstab

When Dynamics 365 is opened directly in a browser, the URL may look like:

https://org.crm4.dynamics.com/main.aspx?appid=...

If the code is executed at the Teams shell level, the URL may simply be:

https://teams.microsoft.com/v2

This inconsistency makes it difficult to determine whether the app is actually running inside Microsoft Teams.

Using parent.window.location.href

If your JavaScript runs inside an iframe, you might try accessing the parent window:

const url = parent.window.location.href;
console.log(url);

In Microsoft Teams, this often still resolves to:

https://teams.microsoft.com/v2

This behavior is expected because Teams heavily sandboxes embedded content and hides the real navigation context for security reasons.

The Key Indicator: source=teamstab

In Dynamics 365, one of the most reliable indicators that the app is opened inside Microsoft Teams is the presence of the query parameter:

source=teamstab

If this parameter exists in the URL, the app was launched from a Microsoft Teams tab.

Example:

https://org.crm4.dynamics.com/uclient/main.htm?...&source=teamstab

Practical Detection Example

The following JavaScript example shows a defensive approach to detect whether the app is running outside Microsoft Teams:

const url = new URL(parent.window.location.href);

if (
  !url.toString().includes("source=teams") &&
  !url.toString().includes("teamstab")
) {
  console.log("Running outside Microsoft Teams");
} else {
  console.log("Running inside Microsoft Teams");
}

This logic works well for Dynamics 365 JavaScript customizations, web resources, embedded Power Apps, and model-driven app extensions.

Best Practices and Considerations

Avoid relying only on window.location, because in embedded scenarios it often reflects only the iframe URL.

Expect strong isolation when running inside Teams. Access to parent and top window objects may be restricted or normalized.

When available, use official APIs such as the Microsoft Teams JavaScript SDK or Power Apps context APIs. These provide explicit context information instead of relying on URL parsing.

Dynamically Loading an External Library in a Dynamics 365 Web Resource

When working with Microsoft Dynamics 365 / Dataverse, JavaScript web resources are a powerful way to extend form behavior. However, loading large third‑party libraries (like Excel parsers, charting tools, or PDF generators) directly in every context is not always ideal.

In this article, we’ll look at a clean and safe approach to dynamically loading an external JavaScript library only when it’s needed — using a real‑world example with SheetJS.

Why Load Libraries Dynamically?

There are several reasons to avoid bundling or always loading external libraries:

• Performance – Large libraries slow down form load times

• Context awareness – Some clients (like Microsoft Teams) have limitations

• Reusability – Load the library only when required

• Maintainability – Easily update the CDN version without redeploying web resources

Dynamic loading solves all of these problems.

The Scenario

We want to:

• Execute code on form OnLoad

• Detect the current client (Web, Outlook, Teams)

• Load SheetJS (xlsx) only if the form is not running inside Microsoft Teams

The OnLoad Function

Here is the full JavaScript example used in a Dynamics 365 form web resource:

OnLoad: function (executionContext) {
    var formContext = executionContext.getFormContext();

    var client = Xrm.Utility
        .getGlobalContext()
        .client
        .getClient();

    if (client !== "Mobile") {
        var script = document.createElement("script");
        script.src = "https://cdn.sheetjs.com/xlsx-latest/package/dist/xlsx.full.min.js";
        script.async = true;
        script.defer = true;
        script.onload = function () {
            console.log("SheetJS library loaded successfully");
        };
        document.head.appendChild(script);
    }
}

Step‑by‑Step Breakdown

1. Access the Form Context

var formContext = executionContext.getFormContext();

This is the recommended approach for interacting with the form in modern Dynamics 365.

2. Detect the Client Type

var client = Xrm.Utility.getGlobalContext().client.getClient();

Possible values include:

• Web

• Mobile

• ... 

This allows us to apply conditional logic depending on where the form is running.

3. Create the Script Element

var script = document.createElement("script");

This uses the browser’s native DOM API to inject JavaScript dynamically.

4. Load the Library from a CDN

script.src = "https://cdn.sheetjs.com/xlsx-latest/package/dist/xlsx.full.min.js";

Using a CDN ensures:

• Faster load times

• Automatic version updates

• Reduced web resource size

5. Handle the Load Event

script.onload = function () {
    console.log("SheetJS library loaded successfully");
};

This guarantees that your code only executes after the library is fully available.

6. Append to the Document Head

document.head.appendChild(script);

At this point, the browser downloads and executes the external library.

Best Practices

• Always check the client context

• Load libraries only when necessary

• Use onload callbacks for dependent logic

• Avoid blocking scripts on form load

• Don’t assume availability in Teams or Mobile clients

When Should You Use This Pattern?

Dynamic loading is ideal when:

• Exporting data to Excel

• Generating documents

• Using visualization libraries

• Adding advanced parsing or calculation logic

PowerCRM SideKick 365: A Must-Have Chrome Extension for Dynamics 365 Professionals

If you work daily with Microsoft Dynamics 365, you know how time-consuming it can be to debug forms, inspect fields, test records, or navigate between system components. PowerCRM SideKick 365 is a Chrome extension designed to significantly improve productivity for developers, testers, and functional consultants.

Download the extension here:
https://chromewebstore.google.com/detail/powercrm-sidekick-365/fjpbmdkkpoioonhibabipdbcohkcdccd

What is PowerCRM SideKick 365?

PowerCRM SideKick 365 adds a powerful side panel directly into your browser when working with Dynamics 365. It provides quick access to tools that normally require multiple clicks, advanced configuration, or external utilities.

Key Features

• Form Tools – Show or hide logical names, enable or disable fields, auto-populate fields with random data

• Update Records – Update any field on a record, including bulk updates

• All Attributes Viewer – Instantly display all attributes of the current entity

• Related Records Explorer – View and navigate all entity relationships

• User Impersonation – Test behavior by impersonating another user

• Plugin Trace Log Explorer – Easily read and analyze plugin trace logs

• Quick Navigation Panel – Jump directly to Solutions, Form Editor, Azure Portal, and more

• Config Manager – Customize which tools appear on different pages

This extension is especially valuable for debugging, testing, and speeding up repetitive CRM tasks without leaving the Dynamics interface.

Similar Chrome Extensions for Dynamics 365 and Power Platform

If you’re building a complete productivity toolkit for Dynamics 365, here are some excellent alternatives and complementary tools.

Toolshed for Power Platform / Dynamics 365
A command-based toolbox that lets you reveal hidden fields, schema names, record IDs, and run quick updates directly from the UI.

XrmWebTools
A web-based extension offering simplified access to XrmToolBox-like features such as audits, plugin logs, and user insights.

CRM 365 Helper
A lightweight extension focused on fast access to common tasks like retrieving record IDs, opening Advanced Find, workflows, and form metadata.

365 Power Pane
Provides basic form manipulation tools and shortcuts, useful for quick inspections and simple testing scenarios.

Each of these tools addresses slightly different needs, but together they can dramatically reduce development and troubleshooting time.

Security and Best Practices

Chrome extensions often require broad permissions. Before installing any tool in a production or corporate environment, make sure to review requested permissions carefully, check user reviews and update frequency, and align installation with your organization’s IT security policies.

Using trusted tools responsibly ensures productivity gains without compromising data security.

PowerCRM SideKick 365 stands out as one of the most complete Chrome extensions for Dynamics 365 professionals. Its integrated side panel and rich feature set make it a powerful companion for everyday CRM development and testing.

Try it here:
https://chromewebstore.google.com/detail/powercrm-sidekick-365/fjpbmdkkpoioonhibabipdbcohkcdccd

If you’re serious about optimizing your Dynamics 365 workflow, combining this extension with tools like Toolshed and XrmWebTools can turn your browser into a full-featured CRM productivity hub.

How to Download the Plugin Registration Tool for Dynamics 365 Using PowerShell (and Key Risks to Consider)

 

If you work with Dynamics 365 CRM / Dataverse, sooner or later you’ll need the Plugin Registration Tool (PRT). This tool is essential for registering, updating, and managing plugins in your Dynamics environment.
In this article, we’ll see how to download it using PowerShell and highlight some important risks and best practices every developer should be aware of.

Why the Plugin Registration Tool Is Important

Dynamics 365 does not provide a built-in UI for managing plugins. The Plugin Registration Tool allows you to:

• Register plugin assemblies

• Add and configure plugin steps

• Define execution pipelines and filtering attributes

• Debug and update existing plugin logic

Without this tool, managing plugins becomes nearly impossible in real-world projects.

Downloading the Plugin Registration Tool Using PowerShell

Instead of downloading executables from random sources, the safest approach is to get the tool directly from Microsoft’s official NuGet packages.

Open PowerShell as Administrator

Make sure PowerShell is running with administrator privileges, otherwise some commands may fail.

Download the NuGet executable

Run the following script to download nuget.exe locally:

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$sourceNugetExe = "https://dist.nuget.org/win-x86-commandline/latest/nuget.exe"
$targetNugetExe = ".\nuget.exe"
Remove-Item .\Tools -Force -Recurse -ErrorAction Ignore
Invoke-WebRequest $sourceNugetExe -OutFile $targetNugetExe
Set-Alias nuget $targetNugetExe -Scope Global -Verbose

This ensures you are using the latest and secure NuGet client.

Install the Plugin Registration Tool

Now download the Plugin Registration Tool package and extract it into a dedicated folder:

./nuget install Microsoft.CrmSdk.XrmTooling.PluginRegistrationTool -O .\Tools
md .\Tools\PluginRegistration
$prtFolder = Get-ChildItem ./Tools | Where-Object {$_.Name -match 'Microsoft.CrmSdk.XrmTooling.PluginRegistrationTool.'}
move .\Tools\$prtFolder\tools\*.* .\Tools\PluginRegistration
Remove-Item .\Tools\$prtFolder -Force -Recurse

Tool Ready to Use

You’ll now find the executable inside the .\Tools\PluginRegistration folder. From there, you can launch the Plugin Registration Tool and connect to your Dynamics 365 environment.

Risks, Warnings, and Best Practices

Avoid Unofficial Downloads

Downloading the Plugin Registration Tool from unofficial websites or shared ZIP files can be risky. These files may contain outdated binaries, modified or malicious code, or incompatible versions that break your environment.
Always prefer PowerShell with NuGet or other Microsoft-supported methods.

Version Compatibility Issues

Using an outdated or incompatible version of the Plugin Registration Tool can lead to connection and authentication errors, failed plugin registrations, unexpected behavior when updating assemblies, or missing and corrupted plugin steps.
This is especially critical when working across different Dynamics 365 or Dataverse versions.

Recommended Best Practices

• Always test plugin changes in a development or sandbox environment

• Use source control systems such as Git or Azure DevOps

• Version your plugin assemblies properly

• Back up existing plugin registrations before making changes

• Consider modern tooling like Power Platform CLI (pac tool) for better ALM integration

What If Eyes Evolved Differently? Inside MIT’s New AI Vision Evolution Project

What if we could replay evolution and explore how eyes might have developed under different environmental pressures? That’s exactly what a team of researchers from MIT, Rice University, and Lund University set out to do with their groundbreaking “What if Eye…?” project — a computational framework that recreates vision evolution inside a virtual world. (eyes.mit.edu)

A Digital Sandbox for Evolution

Instead of waiting millions of years, this project places embodied AI agents — digital creatures inside simulated physics environments — through artificial evolution. They start with a basic light-sensing cell and, over many generations, their visual systems and behaviors evolve in response to survival challenges. The key idea is to let vision emerge naturally from interaction with the environment rather than being manually designed with fixed datasets or human bias. (eyes.mit.edu)

Different Tasks Lead to Different Eyes

The researchers posed “what-if” scenarios by assigning specific tasks to these agents:

• Navigation tasks, like moving through a maze, favor eyes that cover a wide field with many simple sensors — similar to compound eyes seen in insects.

• Detection tasks, such as distinguishing food from poison, drive the evolution of high-resolution, camera-like eyes with a focused forward gaze. (eyes.mit.edu)

This shows that the function of vision directly influences the type of visual system that evolves. (eyes.mit.edu)

Optics Emerge Naturally

One of the most fascinating outcomes is how optical features like lenses emerge in the simulations. When the artificial evolutionary system was allowed to evolve optical genes, it developed lens-like structures — not because they were programmed in, but because they offered a functional advantage. Lenses helped agents overcome the basic trade-off between collecting enough light and maintaining detailed spatial vision. (eyes.mit.edu)

Scaling Brains and Vision Together

The research also found that simply increasing the size of an agent’s “brain” didn’t always make it better at visual tasks. Improvements came only when neural processing and visual acuity scaled together — revealing a close link between sensory input quality and cognitive processing capacity. (eyes.mit.edu)

Beyond Biology — Designing Better Vision Systems

This computational approach doesn’t just help us understand how natural vision might have evolved differently; it also points toward new ways to design artificial vision systems. By treating embodied AI as a “hypothesis-testing machine,” the project lays the foundation for creating bio-inspired sensors and cameras that are optimized for specific tasks, from robotics and drones to wearable devices. (eyes.mit.edu)

How to Import Wikipedia Tables into Google Sheets Using IMPORTHTML

Google Sheets offers a powerful yet simple way to import data directly from the web. One of the most useful functions for this purpose is IMPORTHTML, which allows you to extract tables and lists from web pages such as Wikipedia. In this article, we’ll look at how to use this function step by step, with a practical example.

What Is the IMPORTHTML Function?

The IMPORTHTML function in Google Sheets lets you pull structured data from a webpage and display it automatically in your spreadsheet. This is especially helpful when working with public datasets, statistics, or frequently updated information.

The basic syntax is:

=IMPORTHTML("URL", "table", index)

• URL – The web page you want to extract data from

• "table" or "list" – The type of data you want to import

• index – The number of the table or list on that page

Example: Importing Demographic Data from Wikipedia

Let’s say you want to import demographic data from Wikipedia into Google Sheets. For example, data from the Demographics of India page.

You can use the following formula:

=IMPORTHTML("http://en.wikipedia.org/wiki/Demographics_of_India", "table", 4)

This formula tells Google Sheets to:

• Visit the Wikipedia page

• Look for tables on the page

• Import the fourth table it finds

Once you press Enter, Google Sheets will automatically load the data into your spreadsheet.

Why Use IMPORTHTML?

Here are a few key benefits:

• Automatic updates – When the source page changes, your data updates too

• No manual copy-paste – Saves time and reduces errors

• Perfect for analysis – Ideal for charts, reports, and dashboards

• Beginner-friendly – No coding skills required

Tips for Better Results

• If the imported table isn’t the one you want, try changing the index number

• Make sure the webpage is public and not blocked

• Use additional Google Sheets functions (like QUERY, FILTER, or SORT) to clean and analyze the imported data

The IMPORTHTML function is an excellent tool for bloggers, researchers, students, and analysts who want to work with live web data. With just one formula, you can transform publicly available information into a dynamic and useful spreadsheet.

SQLAlchemy Data wrangling

When working with data, especially at scale, spreadsheets can quickly reach their limits. This is where databases come in.

A database is an organized collection of structured data designed to make data storage, retrieval, modification, and deletion efficient and reliable. Databases are a foundational tool in data analysis, data engineering, and software development.

Why use databases for data wrangling?

Databases offer several key advantages when handling data. They are fast and optimized for performance, even when working with large datasets. They provide administrative features such as access controls, which help protect sensitive information. They also enforce data integrity rules, ensuring that data is entered in the correct format, which is essential for reliable data wrangling.

Types of databases

Databases generally fall into two main categories: relational and non-relational. Relational databases are the most popular and widely used.

Relational databases organize data into tables made up of rows and columns, similar to Excel spreadsheets, but with stricter rules. Each column must have a unique name, all values in a column must share the same data type, and using clear, descriptive column names is considered best practice.

Common examples of relational databases include SQLite, PostgreSQL, MySQL, and SQL Server.

Non-relational databases, often referred to as NoSQL databases, are designed for more flexible or unstructured data, such as documents or key-value pairs. While powerful, they are outside the scope of this introduction.

What is SQL?

SQL, or Structured Query Language, is the standard language used to interact with relational databases. SQL allows users to read, manipulate, and modify data efficiently.

Some common SQL commands include CREATE, which creates a new table in a database; DROP TABLE, which removes a table; SELECT, which retrieves data that matches specific conditions and is also known as a query; and FROM, which specifies the table from which the data should be retrieved.

Using SQL with Python

In Python, libraries such as SQLAlchemy make it easy to work with databases. SQLAlchemy allows you to connect to a database, execute SQL queries, load the results into a pandas DataFrame, and store processed data back into the database.

This creates a powerful workflow that combines the speed and reliability of SQL with the simplicity and flexibility of pandas.

Important things to remember

It is possible to work directly with databases using tools like sqlite3 without SQLAlchemy. SQLAlchemy also supports many database systems beyond SQLite, including PostgreSQL.

Finally, data wrangling does not always have to be done in pandas. In many professional environments, data cleaning and transformation are performed directly in SQL. The best tool often depends on company infrastructure, performance needs, and scalability requirements.

Understanding databases and SQL is a fundamental skill for anyone working with data. Even if pandas is your primary tool, knowing how databases work and how to query them effectively will make you a stronger and more versatile data professional.

BSON: The Binary Format Behind MongoDB

BSON (Binary JSON) is a binary serialization format designed to represent JSON-like data structures in a more efficient, extensible, and performance-oriented way. It is best known as the native data format used by MongoDB, one of the most popular NoSQL databases in the world.

What Is BSON?

BSON was created as an evolution of JSON. While JSON is simple, human-readable, and widely used in web applications, it has limitations when used for high-performance data storage and processing. BSON addresses these limitations by encoding JSON-like documents into a binary representation, while preserving a familiar structure.

In short, BSON is optimized for machine efficiency rather than human readability.

Key Features

BSON offers several important features:

• Binary format: enables faster data parsing and traversal.

• Extended data types: supports additional types such as Date, Binary, ObjectId, Decimal128, and embedded documents.

• Self-describing structure: each element includes type and length information, making parsing efficient.

• Support for complex data models: ideal for hierarchical and semi-structured data.

BSON and MongoDB

MongoDB uses BSON as its internal data storage format. This design allows the database to:

• efficiently index document fields;

• execute fast queries on nested and complex data;

• handle large volumes of unstructured or semi-structured data.

When developers interact with MongoDB using JSON through APIs or drivers, the conversion to BSON happens automatically behind the scenes.

 JSON vs BSON

Text-based	Binary
Human-readable	Not human-readable
Limited data types	Rich data types
Smaller in some cases	Faster to process

Although BSON documents can be larger in size than JSON equivalents, this trade-off is justified by improved performance and greater flexibility.

Advantages and Disadvantages

Advantages

• High performance

• Rich data typing

• Well-suited for NoSQL databases

Disadvantages

• Not directly readable by humans

• More complex than JSON

• Requires specific libraries to parse

BSON is a powerful and efficient data representation format designed for modern database systems. While it is rarely used directly by developers, it plays a critical role in enabling MongoDB’s performance, scalability, and flexibility. Understanding BSON provides valuable insight into how NoSQL databases manage and process data internally.

Web Scraping: What It Is, When to Use It, and Why to Do It Ethically

In the world of data, one of the most common challenges is accessing information. Very often, the data we need is not available in downloadable formats (such as CSV or Excel), but is instead embedded within web pages. This is where web scraping comes into play.

What Is Web Scraping?

Web scraping is a technique that allows you to extract data from websites using code. Web pages are written in HTML (HyperText Markup Language), a language that uses tags to structure content (headings, paragraphs, tables, links, etc.).

Since HTML is essentially text, it can be read and analyzed by programs called parsers, which make it possible to automatically locate and retrieve the desired information.

In practice, instead of manually copying and pasting data from a website, we can write a script that does it for us.

How Do You Obtain HTML Data?

HTML data can be collected in two main ways:

• By manually downloading the HTML source code of a web page

• By programmatically accessing the website via HTTP requests (for example, using a GET request)

Once the HTML is obtained, it can be analyzed and transformed into structured data ready for analysis.

When Not to Use Web Scraping

It’s important to clarify a crucial point: web scraping is not always allowed.

Many websites impose specific restrictions in their Terms and Conditions, and ignoring them can lead to legal issues. For this reason, it’s essential to do your homework before starting any scraping activity.

Here are some fundamental guidelines to follow:

• Always check the website’s Terms and Conditions

• Consider whether your data usage is personal, academic, or commercial

• Act ethically and responsibly

• If the website offers a public API, use it instead of scraping

• Send HTTP requests at a reasonable frequency

• Avoid massive or simultaneous requests that could resemble a DDoS attack

• Stay informed about laws and regulations related to web scraping

Web scraping is not just a technical matter, but also an ethical one. There are excellent articles that explore this topic further, such as “Ethics in Web Scraping” on Towards Data Science.

API vs Web Scraping

Whenever possible, it’s always better to choose APIs over scraping.

Why?

• APIs are more stable: they don’t depend on a website’s layout

• They are specifically designed to provide data

• They offer data that is already structured and easy to use

• They scale better with increased request volume

Web scraping, on the other hand, is fragile: even a small change in the website’s HTML code (a redesign, a new tag, a different class) can completely break your script.

Golden rule: if an official API exists, use it.

Key Terms to Know

For beginners, here are some essential concepts:

• HTML (HyperText Markup Language): the markup language used to create web pages

• Parser: a tool that analyzes HTML code to extract information

• Web Scraping: a technique for extracting data from websites using code

Web scraping is a powerful and highly useful tool for anyone working with data, but it must be used consciously and responsibly. Understanding how HTML works, when to use scraping, and when to avoid it is essential to becoming an effective—and above all ethical—data wrangler.

If you want to work with web data, remember: respect websites, respect the rules, and always choose the best solution between scraping and APIs.