" MicromOne

Pagine

Stub Testing a Complete Guide to Reliable and Maintainable Software Tests

 Software testing is one of the foundations of reliable application development. As applications become increasingly complex, however, testing every component against real external dependencies can make test suites slow, fragile, and difficult to maintain.

This is where stub testing becomes extremely useful.

A stub allows developers to replace a real dependency with a controlled substitute that returns predefined responses. Instead of contacting a real database, external API, file system, or third-party service, a test can work with predictable data and focus on the behavior of the component being tested.

In this guide, we will explore what stub testing is, how it differs from mocks and fakes, how to design effective stubs, and how to use them in popular programming environments such as Python, JavaScript, and Java.

What Is Stub Testing?

A stub is a test double that provides predefined responses when it is called by the system under test.

For example, imagine an application that retrieves a user's profile from an external API. During a unit test, contacting the real API would introduce an unnecessary external dependency.

Instead, we can replace the API client with a stub that always returns a known user profile.

The test can therefore verify the application's logic without making a real network request.

This approach provides several important advantages:

  • Speed: tests avoid slow external services.

  • Reliability: external outages do not cause test failures.

  • Determinism: the same input produces predictable results.

  • Isolation: developers can test one component independently.

  • Flexibility: unusual or difficult scenarios can be simulated easily.

The main principle is simple: replace an external dependency with a controlled response so that the component under test can be evaluated in isolation.

Stub vs. Mock vs. Fake

Stub, mock, and fake are all examples of test doubles, but they serve different purposes.

Stub

A stub primarily provides predefined data or behavior.

For example:

When getUser(1) is called,
return { id: 1, name: "Alice" }

The goal is to provide the data required by the test.

Mock

A mock is generally used when the test also needs to verify interactions.

For example, a mock can verify that:

  • a method was called;

  • it was called with specific parameters;

  • it was called a particular number of times;

  • calls occurred in a specific order.

Mocks are therefore particularly useful when the interaction between components is itself part of the behavior being tested.

Fake

A fake is a simplified working implementation of a real component.

A common example is an in-memory database used instead of a production database. Unlike a simple stub, a fake may contain actual logic and state.

In practice, the choice depends on the objective of the test:

Test doubleMain purpose
StubProvide predefined responses
MockVerify interactions
FakeProvide a simplified working implementation

Keeping these concepts separate can make a test suite easier to understand and maintain.

Why Use Stub Testing?

One of the biggest advantages of stub testing is isolation.

Consider an application that calculates information using data retrieved from an external service. If the test depends directly on that service, a failure could have many possible causes:

  • the external service is unavailable;

  • the network connection fails;

  • the service returns unexpected data;

  • authentication expires;

  • the API changes;

  • the response takes too long.

None of these problems necessarily indicate a bug in the application code being tested.

A stub removes those variables.

The test can provide exactly the response required and concentrate on the application's behavior.

This makes failures easier to understand and significantly improves the reliability of automated testing.

How to Design an Effective Stub Test

Creating a good stub is not simply a matter of replacing every dependency with fake data. The stub should support the objective of the test without making the test unnecessarily complicated.

Define the Test Objective

Before creating a stub, determine exactly what you want to test.

Identify:

  • the component under test;

  • its inputs;

  • its expected outputs;

  • the dependency that needs to be replaced;

  • the scenarios that must be simulated.

A well-defined objective prevents the stub from becoming more complicated than the code it is testing.

Identify the Right Dependencies

Not every dependency needs to be stubbed.

Focus on components that are:

  • external;

  • slow;

  • unreliable;

  • expensive;

  • difficult to reproduce;

  • dependent on network resources;

  • dependent on unpredictable data.

Typical examples include APIs, databases, message queues, file systems, and third-party services.

Make Responses Deterministic

A good stub should behave predictably.

For example, if a test needs to verify successful authentication, the stub should consistently return a successful response.

You should also consider negative and edge cases, such as:

  • HTTP errors;

  • missing data;

  • invalid input;

  • timeouts;

  • empty responses;

  • unexpected values.

This makes it possible to test scenarios that may be difficult to reproduce with a real service.

Keep Stub Configuration Simple

If the same stub is used by multiple tests, consider centralizing its configuration.

A factory, helper function, fixture, or dedicated test utility can prevent duplicated configuration and make future changes easier.

Keep Stubs Updated

External interfaces evolve.

If an API changes its response format, the corresponding stubs may also need to be updated.

Outdated stubs can create a dangerous situation where tests continue to pass even though they no longer accurately represent real application behavior.

For this reason, stub maintenance should be treated as part of normal software maintenance.

Stub Testing in Python

Python provides several options for working with test doubles. The standard library includes unittest.mock, which can be used to replace dependencies during testing.

For example, imagine a function that retrieves a user profile:

def fetch_user_profile(user_id, api_client):
    response = api_client.get(
        f"https://api.example.com/users/{user_id}"
    )

    if response.status_code == 200:
        return response.json()

    raise ValueError("Unable to retrieve user profile")

A test can provide a controlled API client instead of making a real HTTP request:

def test_fetch_user_profile():
    class ApiStub:
        def get(self, url):
            return type(
                "Response",
                (),
                {
                    "status_code": 200,
                    "json": lambda self: {
                        "id": 1,
                        "name": "Alice"
                    }
                }
            )()

    result = fetch_user_profile(1, ApiStub())

    assert result == {
        "id": 1,
        "name": "Alice"
    }

The important idea is that the function does not know whether it is communicating with the real API or the test implementation.

The test therefore remains fast and deterministic.

Stub Testing in JavaScript

JavaScript and TypeScript developers frequently use testing frameworks such as Jest.

Consider a function that calculates a value using a rate retrieved from an external service:

export async function computeTax(income, api) {
    const rate = await api.getTaxRate();
    return income * rate;
}

A simple stub can provide a predictable tax rate:

test("computeTax uses the rate supplied by the stub", async () => {
    const apiStub = {
        getTaxRate: async () => 0.20
    };

    const tax = await computeTax(50000, apiStub);

    expect(tax).toBe(10000);
});

The test does not need to access a real external service.

This makes the test faster and ensures that the calculation is evaluated using known data.

Stub Testing in Java

Java developers can implement stubs manually or use libraries such as Mockito.

For example:

TaxApi apiStub = mock(TaxApi.class);

when(apiStub.getTaxRate()).thenReturn(0.15);

TaxService service = new TaxService(apiStub);

double result = service.calculateTax(40000);

assertEquals(6000, result, 0.001);

Here, getTaxRate() always returns the predefined value.

Mockito can also be used to verify interactions when the test needs to confirm that a particular method was called.

This illustrates an important distinction: a test may use a stub to provide data and mock-style verification to check interactions.

Common Mistakes When Using Stubs

Stub testing is powerful, but excessive or inappropriate use can reduce the quality of a test suite.

Overusing Stubs

If every dependency is replaced with a stub, tests can become disconnected from reality.

A test may pass even though the real components do not work correctly together.

Using Unrealistic Data

Stub responses should resemble realistic production scenarios.

If real APIs can return missing fields, errors, delays, or unexpected values, those scenarios should be represented in appropriate tests.

Forgetting Integration Tests

Unit tests with stubs are excellent for isolation, but they cannot completely replace integration testing.

Real integrations should still be tested in controlled environments to verify that components communicate correctly.

Creating Overly Complex Stubs

A stub should normally be simpler than the component it replaces.

If a stub contains complicated business logic, it may become another source of bugs and make tests harder to understand.

Best Practices for Maintainable Stub Tests

A few principles can make stub-based testing considerably more effective.

Use Clear Names

Names such as:

apiClientStub
databaseStub
paymentServiceStub
userRepositoryStub

make the purpose of each test double immediately understandable.

Test Both Success and Failure

Do not only simulate successful responses.

Include scenarios such as:

  • invalid requests;

  • missing resources;

  • server errors;

  • empty results;

  • timeouts;

  • malformed data.

Keep Tests Focused

Each test should have a clear purpose.

Avoid creating enormous test scenarios involving multiple unrelated stubs and complex configurations.

Review Stubs Regularly

As the production code and external services evolve, review the corresponding stubs.

Remove obsolete test doubles and update responses when interfaces change.

Combine Unit and Integration Testing

The strongest testing strategy usually combines different levels of testing.

Unit tests can use stubs to provide fast feedback, while integration tests can verify that real components work together correctly.

When Should You Use Stub Testing?

Stub testing is particularly useful when a component depends on something outside its direct control.

Typical examples include:

  • REST APIs;

  • payment services;

  • databases;

  • cloud services;

  • authentication providers;

  • file systems;

  • message queues;

  • external APIs;

  • third-party libraries.

A stub is especially valuable when the external dependency is slow, expensive, unreliable, or difficult to reproduce.

However, the objective should not be to eliminate every real dependency from testing.

Instead, use stubs where they provide meaningful isolation and combine them with integration and end-to-end tests where real system behavior needs to be validated.

Stub testing is a simple but powerful technique for building reliable automated tests.

By replacing complex or unpredictable dependencies with controlled responses, developers can create tests that are faster, more deterministic, and easier to troubleshoot.

The key is to use stubs strategically.

A good stub should be simple, predictable, realistic enough for the scenario being tested, and easy to maintain. At the same time, stub-based unit tests should complement—not replace—integration and end-to-end testing.

Whether you work with Python, JavaScript, Java, or another programming language, understanding how to use stubs effectively can significantly improve the quality and maintainability of your testing strategy.

Isolate what you need to test, control what you need to simulate, and keep your test doubles as simple as possible.

The Power of Active Listening Why Great Teams Listen Before They Act


Whether you are an athlete, coach, captain, or team manager, effective communication starts with one simple skill: listening.

Listening to teammates, coaches, opponents, and staff helps us understand different perspectives, identify needs, build trust, and create stronger relationships.

But there is a big difference between simply hearing someone and truly listening to them.

That is where active listening comes in.

What Is Active Listening?

Active listening means being fully engaged in a conversation.

It is not just about hearing the words someone says. It is about understanding their message, emotions, intentions, and point of view.

In sport, communication is often about much more than words. A player's body language, tone of voice, facial expressions, and attitude can communicate just as much as what they actually say.

An actively engaged listener gives the speaker their full attention. That means putting distractions aside, making eye contact, and being present in the moment.

Most importantly, active listening means trying to understand someone else's perspective without immediately judging it or imposing your own opinion.

You do not have to agree with your teammate or coach. But before responding, you should make sure you understand where they are coming from.

Why Does Active Listening Matter in Sport?

Active listening can have a powerful impact on both individual performance and team culture.

It Builds Trust and Stronger Relationships

Trust is one of the foundations of successful teams.

When athletes feel that their teammates and coaches genuinely listen to them, they are more likely to feel respected and valued. This creates an environment where people feel comfortable expressing concerns, sharing ideas, and asking for help.

For coaches, listening can also provide valuable insight into what athletes are experiencing, both on and off the field.

It Improves Understanding

A disagreement between teammates is not always about the issue itself. Sometimes, the real problem is that people have different experiences or expectations.

Active listening helps uncover the context behind someone's opinion.

Instead of simply thinking, "I disagree with you," an athlete can ask, "Why do you see it that way?"

That small change can lead to a much deeper understanding and prevent unnecessary conflict.

It Can Improve Team Performance

Sport is fast-paced. Players need to understand instructions, adapt to changing situations, and respond quickly to their teammates.

Good listening helps make communication more efficient.

When players listen carefully during training, meetings, tactical discussions, and feedback sessions, they are more likely to understand what is expected of them and execute their role correctly.

In other words, better listening can mean fewer misunderstandings and fewer mistakes.

How to Become a Better Listener

Active listening is a skill that can be developed with practice. Here are some simple techniques athletes and coaches can use.

Make Eye Contact

When someone is speaking to you, look at them and show that you are engaged.

You do not need to stare constantly, but appropriate eye contact and an open posture can communicate that you are paying attention.

Show Encouragement

Small reactions can reassure the other person that you are following the conversation.

A nod, a smile, or a simple "yes," "I understand," or "that makes sense" can encourage someone to continue speaking.

These small signals can be particularly important when discussing difficult topics.

Don't Interrupt

Give people the opportunity to finish their thoughts.

In a competitive sporting environment, it can be tempting to respond immediately, defend yourself, or explain your position. But interrupting can prevent you from hearing the complete message.

Let the person finish. Then respond.

Ask Open-Ended Questions

If something is unclear, ask for clarification.

Instead of making assumptions, use open-ended questions that encourage the other person to explain their perspective.

For example:

"Can you tell me more about what happened?"

"What did you feel was missing from the team's approach?"

The goal is not to prove someone wrong. The goal is to understand.

Paraphrase What You Heard

One of the most effective active-listening techniques is to repeat the message in your own words.

You might say:

"Just to make sure I understand, you're saying that you felt we didn't communicate well during the second half. Is that right?"

This gives the other person the opportunity to confirm what you understood or correct you if necessary.

It can prevent misunderstandings before they become bigger problems.

Active Listening Starts With Empathy

Perhaps the most important part of active listening is empathy.

In sport, everyone sees situations through their own experience.

A coach may focus on tactics and performance.

An athlete may be thinking about confidence, fatigue, motivation, or pressure.

A captain may be concerned about team chemistry.

Active listening allows us to temporarily step outside our own perspective and consider what someone else might be experiencing.

That does not mean abandoning your own opinion. It means making space for another person's point of view before deciding how to respond.

From Communication to Teamwork

Great teams are not built only through talent and physical preparation. They are built through trust, respect, understanding, and communication.

And communication begins with listening.

When athletes listen to one another, coaches listen to their players, and leaders listen to their teams, misunderstandings become easier to resolve and relationships become stronger.

The next time you are in a team meeting, training session, locker room, or difficult conversation, try something simple: listen before you respond.

You may discover that the most important part of communication is not having the right answer, but making sure you truly understand the person speaking.

The Art of Persuasion How to Influence, Inspire and Lead a Team



Behind every great performance, there is communication, leadership, trust, and the ability to bring people together around a common goal.

Coaches need to convince athletes to embrace a tactical approach. Captains need to motivate teammates. Players need to communicate ideas and influence decisions on and off the field.

This is where persuasion becomes an essential skill.

Persuasion is not about forcing people to agree with you. It is about presenting your ideas in a way that helps others understand your perspective, recognize its value, and ultimately move toward a shared objective.

Why Does Persuasion Matter in Sport?

A sports team is made up of different personalities, experiences, motivations, and opinions.

A coach may have a clear vision for how the team should play, but simply giving instructions does not guarantee that players will fully understand or believe in them.

The same applies to team captains. A captain does not always have formal authority over teammates. Leadership often comes from influence rather than position.

To create alignment, you need to communicate your ideas and persuade others to believe in the direction you are proposing.

Whether the goal is changing a tactical approach, improving team culture, or preparing for an important competition, persuasion can help turn individual opinions into collective action.

The Three Elements of Persuasion

More than 2,000 years ago, Greek philosopher Aristotle identified three fundamental modes of persuasion:

  • Ethos — credibility

  • Pathos — emotion

  • Logos — logic

These principles remain highly relevant in modern sport.

Ethos: Build Credibility

Ethos is about credibility and trust.

In sport, people are more likely to listen to someone they respect and trust. A coach earns credibility through experience, knowledge, consistency, and the ability to make sound decisions.

Athletes can also develop credibility through their performance, character, professionalism, and commitment to the team.

However, credibility is not built overnight.

It comes from consistently:

  • Doing what you say you will do

  • Supporting your teammates

  • Demonstrating professionalism

  • Making decisions based on the team's best interests

Pathos: Connect Through Emotion

Sport is emotional.

Confidence, fear, excitement, disappointment, pride, and motivation can all influence performance.

Pathos is the emotional side of persuasion. It is about understanding how you want your audience to feel and communicating in a way that creates that emotional connection.

Think about a coach speaking to a team before a championship game.

Statistics and tactics matter, but sometimes the message that players remember is the one that makes them believe in themselves and in each other.

A strong story, personal experience, or even a well-timed moment of humor can make a message more memorable and help people connect with it.

Logos: Use Logic and Evidence

Logos is the logical side of persuasion.

In sport, this often means using:

  • Statistics

  • Performance data

  • Tactical analysis

  • Evidence

  • Previous game results

For example, instead of simply telling a team that a particular strategy will work, a coach can demonstrate how the strategy has performed in previous games or how the team's data supports the decision.

Data does not need to overwhelm the audience. A few relevant statistics can often be more powerful than a presentation filled with numbers.

The Best Persuasive Messages Combine All Three

Ethos, pathos, and logos should not be treated as separate tools.

The most effective communication combines credibility, emotion, and logic.

A coach with strong credibility can make statistical information more convincing. A powerful statistic can create an emotional reaction. A personal story can strengthen the connection between a leader and the team.

Together, these elements create a much stronger message.

Know Your Audience

Before trying to persuade someone, understand who you are speaking to.

A message that motivates one athlete may have little impact on another.

Some players respond to statistics and technical explanations. Others are motivated by personal challenges, team goals, or emotional encouragement.

Understanding what your audience values allows you to adapt your communication without changing the fundamental message.

The key question is:

What matters to the people I am trying to influence?

Once you know the answer, you can communicate your idea in a way that feels relevant to them.

Choose a Clear Point of View

One common mistake in communication is trying to present every possible option without making a recommendation.

In sport, this can create confusion.

If a coach believes the team should change its defensive strategy, the players need to understand what the recommended approach is and why it is considered the best option.

That does not mean ignoring alternative perspectives.

A strong leader should understand different options, including their advantages and disadvantages, and be prepared to discuss them.

But ultimately, effective persuasion requires clarity and decisiveness.

Support Your Ideas With Data

Opinions are important, but evidence makes arguments stronger.

Performance data can help coaches and athletes move beyond personal preferences and focus on what is actually happening.

For example, statistics can reveal:

  • Where the team is losing possession

  • Which areas of the field create the most opportunities

  • How effectively players are pressing

  • Where defensive problems are occurring

  • Which tactical choices are producing better results

The goal is not to use data simply to prove that you are right.

The goal is to use evidence to help the team make better decisions.

Don't Ignore Objections

When someone disagrees with you, it can be tempting to dismiss their argument.

That is rarely effective.

Ignoring concerns can make you appear defensive and can damage trust within the team.

Instead:

  • Listen to objections

  • Discuss them openly

  • Ask questions

  • Explore different perspectives

  • Consider the advantages and disadvantages of each option

  • Give people the opportunity to explain why they disagree

This approach demonstrates that you are not simply trying to win an argument. You are trying to find the best solution for the team.

And sometimes, that solution will not be your original idea.

Great leaders understand that changing their mind when presented with better information is not weakness. It is good leadership.

Always Finish With a Call to Action

A persuasive conversation should lead somewhere.

Once everyone understands the argument and the group has reached a decision, make sure the next steps are clear.

In a sports environment, that might mean:

"Starting tomorrow, we'll use this defensive system in training. Everyone will have a specific role, and we'll review the results after the next two sessions."

Clear actions turn an idea into a plan.

Without a clear next step, even the most persuasive presentation can lose its impact.

Persuasion Is About Leadership, Not Winning

You will not win every discussion.

And that is perfectly fine.

The goal of persuasion is not to make sure everyone agrees with you every time. It is to create better conversations, encourage understanding, and help people move toward a common objective.

Over time, consistently communicating with credibility, empathy, logic, and clarity builds trust.

Your teammates learn that you listen.

Your athletes learn that your decisions have a purpose.

Your coaches and colleagues learn that you can present a strong argument while still respecting different perspectives.

That is when persuasion becomes more than a communication technique.

It Becomes a Leadership Skill

In sport, the strongest leaders are not necessarily the loudest people in the room.

They are the people who can inspire others to believe in a shared vision — and then turn that belief into action.

Building Passwordless-by-Design Authentication with OPAQUE, Next.js, TypeScript, PostgreSQL, and Docker

Most web applications still authenticate users by sending a password to a server and asking the server to compare it with a stored password hash.

This model is well established, but it has an important property that is easy to overlook:

the authentication server receives the user's password.

Even when TLS is used, the password exists in the server's memory during the authentication request. Password hashing protects stored credentials if a database is stolen, but it does not change the fundamental authentication flow.

There is another approach.

In this project, I built a complete authentication system using OPAQUE, a password-authenticated key exchange protocol designed so that the server never receives the user's password.

The stack is:

  • Next.js

  • TypeScript

  • Node.js

  • PostgreSQL

  • Docker

  • OPAQUE / RFC 9807

The result is a full-stack authentication system in which the password remains on the client while the server still gets cryptographic proof that the user knows it.

Why OPAQUE?

A traditional authentication flow usually looks like this:

Browser                         Server
   |                              |
   | email + password             |
   |----------------------------->|
   |                              |
   | password verification        |
   |                              |
   |<-------- authenticated ------|

TLS protects the connection, but the server still receives the password.

A typical password-hash-based system therefore has two different security boundaries:

  1. The password must be protected while travelling over the network.

  2. The password must be protected while being processed by the server.

OPAQUE changes the model.

Instead of sending the password to the authentication server, the client executes a cryptographic protocol involving an Oblivious Pseudorandom Function (OPRF) and an authenticated key exchange.

Conceptually:

Browser                         Server
   |                              |
   | OPAQUE message               |
   |----------------------------->|
   |                              |
   | cryptographic response       |
   |<-----------------------------|
   |                              |
   | OPAQUE message               |
   |----------------------------->|
   |                              |
   | authenticated session        |

The password itself never crosses the API boundary.

OPAQUE is standardized in RFC 9807.

The Architecture

The project uses Next.js as the application layer, PostgreSQL for persistent storage, and Docker for reproducible development and deployment.

The high-level architecture is:

                     ┌──────────────────────┐
                     │      Browser         │
                     │                      │
                     │ Password             │
                     │ OPAQUE client        │
                     └──────────┬───────────┘
                                │
                         HTTPS / OPAQUE
                                │
                                ▼
                     ┌──────────────────────┐
                     │      Next.js         │
                     │                      │
                     │ Authentication API   │
                     │ OPAQUE server        │
                     │ Session management   │
                     └──────────┬───────────┘
                                │
                                ▼
                     ┌──────────────────────┐
                     │     PostgreSQL       │
                     │                      │
                     │ User records         │
                     │ OPAQUE records       │
                     │ Session hashes       │
                     └──────────────────────┘

The most important design decision is that PostgreSQL does not contain a traditional password hash.

Instead, it stores the OPAQUE registration record generated by the client.

Registration

The registration process starts entirely in the browser.

The user enters:

Email
Password

The password is passed directly to the OPAQUE client implementation.

The browser generates an OPAQUE registration request:

Browser
   │
   │ password
   │
   │ OPAQUE registration
   ▼
RegistrationRequest

The registration request is sent to the server.

The server processes it using its OPAQUE server configuration and returns a registration response.

The browser then completes the registration process and generates the final registration record.

Conceptually:

Browser                              Server

password
   │
   │ startRegistration()
   │
   │ RegistrationRequest
   ├──────────────────────────────────►
   │
   │ RegistrationResponse
   ◄───────────────────────────────────┤
   │
   │ finishRegistration()
   │
   │ RegistrationRecord
   ├──────────────────────────────────►
   │
   │                         PostgreSQL
   │                         stores record

The important detail is what doesn't happen:

POST /register

{
    "email": "...",
    "password": "..."
}

There is no such request.

The API receives OPAQUE protocol data instead.

What Does PostgreSQL Store?

The database contains a user record similar to:

CREATE TABLE users (
    id UUID PRIMARY KEY,
    email TEXT NOT NULL UNIQUE,
    opaque_record BYTEA NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

There is no:

password

column.

There is also no traditional:

password_hash

column.

The opaque_record is cryptographic protocol data that allows the server to participate in future authentication attempts without learning the password.

The Login Flow

The login process is where OPAQUE becomes particularly interesting.

The browser starts an OPAQUE login using the password entered by the user.

The first message is KE1.

Browser                         Server

password
   │
   │ startLogin()
   │
   │ KE1
   ├─────────────────────────────►
   │
   │ KE2
   ◄─────────────────────────────┤
   │
   │ finishLogin()
   │
   │ KE3
   ├─────────────────────────────►
   │
   │ authenticated

The server uses the stored OPAQUE registration record and its server setup to generate the appropriate response.

The browser then completes the protocol.

If the password is correct, both sides derive the appropriate cryptographic session material.

If the password is incorrect, authentication fails.

At no point does the server need to receive the actual password.

OPAQUE Is Not Simply "Encrypting the Password"

This distinction is important.

A tempting implementation would be:

password
   ↓
encrypt
   ↓
send to server

That is not the goal.

The server would still receive encrypted password material, and the overall security properties would depend heavily on key management and the encryption scheme.

Another common idea is:

password
   ↓
SHA-256
   ↓
send hash

This is also not sufficient.

If the server accepts a static password hash as the authentication credential, that hash can effectively become a password-equivalent.

OPAQUE instead uses a password-authenticated key exchange protocol designed specifically for this problem.

Application Sessions

After a successful OPAQUE exchange, the application still needs an HTTP session.

I deliberately keep these two concepts separate.

OPAQUE handles the cryptographic authentication.

The application session handles normal web application authorization.

The flow therefore becomes:

             OPAQUE
Browser ──────────────────► Server
        authentication

             ↓

       authenticated

             ↓

      application session

             ↓

        HttpOnly cookie

The project creates a cryptographically random session token.

The browser receives it through an HttpOnly cookie.

The database does not store the raw token.

Instead, it stores a SHA-256 hash:

random session token
        ↓
      SHA-256
        ↓
    PostgreSQL

This means a database leak does not directly reveal active session tokens.

Session Storage

The session table is intentionally simple:

CREATE TABLE sessions (
    id UUID PRIMARY KEY,
    user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
    token_hash BYTEA NOT NULL UNIQUE,
    expires_at TIMESTAMPTZ NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

The cookie is configured with:

HttpOnly
Secure
SameSite

The Secure flag is enabled in production.

This prevents JavaScript from directly reading the session token and helps reduce the impact of client-side attacks.

Why PostgreSQL?

PostgreSQL is not responsible for the cryptographic protocol.

Its job is persistence.

It stores:

  • user identifiers

  • OPAQUE registration records

  • session hashes

  • expiration information

This separation keeps the architecture straightforward.

The cryptographic protocol runs at the application layer, while PostgreSQL provides durable storage.

Next.js API Routes

The project exposes separate endpoints for the different protocol stages.

Registration starts with:

POST /api/auth/register/start

The browser sends the OPAQUE registration request.

The second stage is:

POST /api/auth/register/finish

The browser sends the generated OPAQUE registration record.

Login follows the same idea:

POST /api/auth/login/start

and:

POST /api/auth/login/finish

There are also endpoints for:

POST /api/auth/logout
GET  /api/auth/me

This makes the authentication protocol explicit instead of hiding everything behind a conventional email + password endpoint.

Docker

The development environment contains two services:

services:

  postgres:
    image: postgres:17-alpine

  web:
    build: .

PostgreSQL is isolated inside its own container.

The Next.js application runs in another container.

This makes the complete environment reproducible:

docker compose up --build

The application becomes available on:

http://localhost:3000

For development, PostgreSQL can also be run separately while Next.js runs directly with Node.js.

Generating the OPAQUE Server Setup

OPAQUE requires server-side cryptographic configuration.

The project includes a helper command:

npm run opaque:setup

The generated value is placed in:

OPAQUE_SERVER_SETUP=...

This value is a secret.

It should never be committed to Git.

In production, it should ideally be stored in a proper secret-management system or KMS rather than in a plain .env file.

Changing the server setup is also not something to do casually: existing OPAQUE records depend on the server configuration.

Rate Limiting

OPAQUE protects the password from being transmitted to the server, but it does not eliminate online password guessing.

An attacker can still repeatedly attempt authentication.

Therefore the API needs rate limiting.

The starter project includes a simple in-memory rate limiter:

IP / account
     ↓
rate limiter
     ↓
OPAQUE authentication

For production, this should be replaced with a distributed mechanism such as Redis.

This becomes particularly important when multiple application instances are running behind a load balancer.

Account Enumeration

Authentication systems often accidentally reveal whether an account exists.

For example:

"Email does not exist"

versus:

"Password is incorrect"

This allows attackers to build lists of valid accounts.

A production implementation should return deliberately generic authentication errors.

OPAQUE implementations can also use a fake record strategy to make nonexistent accounts behave more like existing accounts during the protocol.

This is an important part of designing the complete authentication system rather than simply implementing the cryptographic primitive.

Horizontal Scaling

There is one important limitation in the starter implementation.

During login, the server needs temporary state between the OPAQUE login messages.

The example implementation keeps that state in process memory.

That is fine for:

Browser
   ↓
Server A

but problematic for:

             ┌── Server A
Load Balancer│
             └── Server B

The first request could reach Server A while the second reaches Server B.

For production, the ephemeral OPAQUE state should therefore be stored in a shared system such as Redis.

The architecture becomes:

                 Load Balancer
                       │
              ┌────────┴────────┐
              ▼                 ▼
          Next.js A         Next.js B
              │                 │
              └────────┬────────┘
                       ▼
                     Redis
                 temporary state

The state should have a short expiration time.

Security Headers

The application also sets basic security headers, including:

X-Content-Type-Options
X-Frame-Options
Referrer-Policy
Permissions-Policy
Content-Security-Policy
Strict-Transport-Security

These headers are not part of OPAQUE itself.

They are simply additional layers of web application security.

A secure authentication protocol does not make the rest of the application secure automatically.

What OPAQUE Does Not Solve

OPAQUE is powerful, but it is not magic.

It does not protect against:

  • phishing

  • malware on the user's device

  • compromised browsers

  • weak passwords

  • malicious browser extensions

  • compromised TLS termination

  • stolen authenticated sessions

  • denial-of-service attacks

  • application authorization bugs

For example, if an attacker steals a user's active session cookie, they may be able to use the account without knowing the password.

That's why authentication needs to be considered as a complete security system.

Password Recovery Is Different

One particularly interesting consequence of this architecture is password recovery.

In a conventional application, an administrator can sometimes reset a password hash directly.

With OPAQUE, there is no ordinary password hash that the server can simply replace.

A password reset therefore requires a carefully designed recovery flow.

A production implementation should define how the user proves control of an independent recovery channel, such as an email address, and then establishes a new OPAQUE credential.

This is an important design area and should not be bolted on as an afterthought.

Adding Passkeys

OPAQUE and WebAuthn/passkeys solve related but different problems.

OPAQUE provides strong password-based authentication while keeping the password hidden from the server.

Passkeys eliminate the password from the authentication experience entirely and use public-key credentials.

A mature authentication platform can support both:

                 Authentication
                       │
             ┌─────────┴─────────┐
             ▼                   ▼
          OPAQUE             Passkeys
        password-based       WebAuthn

Users who want password authentication can use OPAQUE, while users with compatible devices can use passkeys.

Project Structure

The final project is organized approximately as follows:

opaque-next-auth/
│
├── app/
│   ├── api/
│   │   └── auth/
│   │       ├── register/
│   │       ├── login/
│   │       ├── logout/
│   │       └── me/
│   │
│   ├── register/
│   ├── login/
│   └── account/
│
├── components/
│
├── lib/
│   ├── db.ts
│   ├── opaque.ts
│   ├── security.ts
│   ├── challenges.ts
│   └── rate-limit.ts
│
├── db/
│   └── 001_init.sql
│
├── scripts/
│
├── Dockerfile
├── docker-compose.yml
├── package.json
└── README.md

The goal is to keep the cryptographic layer, persistence layer, session layer, and UI clearly separated.

Running the Project

After cloning the project:

npm install

Generate the OPAQUE server setup:

npm run opaque:setup

Create the environment file:

cp .env.example .env

Set:

DATABASE_URL=postgres://app:app_dev_password_change_me@localhost:5432/opaque_auth

OPAQUE_SERVER_SETUP=YOUR_GENERATED_SETUP

APP_ORIGIN=http://localhost:3000

Start PostgreSQL:

docker compose up -d postgres

Run the migration:

npm run db:migrate

Then start Next.js:

npm run dev

Open:

http://localhost:3000

You can then create an account and inspect the network requests.

The most interesting thing to observe is that there is no HTTP request containing the user's password.

The most interesting aspect of OPAQUE is not simply that it uses advanced cryptography.

It changes the security boundary.

In a conventional password authentication system, the authentication server is trusted with the password during the login process.

With OPAQUE, the server can authenticate the user without receiving the password itself.

That does not eliminate every authentication risk, but it removes an important piece of sensitive information from the server-side application layer.

For applications where password privacy and resistance to server-side credential exposure are important, OPAQUE is therefore a technology worth considering.

The project described here provides a practical starting point using:

Next.js
   +
TypeScript
   +
OPAQUE
   +
PostgreSQL
   +
Docker

The next step toward a production deployment would be to add distributed challenge storage, robust rate limiting, account recovery, email verification, comprehensive integration tests, monitoring, secret management, and an independent security review.

Cryptography can provide strong primitives.

The real security of the system, however, still depends on how those primitives are integrated into the rest of the application.

OpenAI’s Jalapeño Chip Delivers Faster, More Efficient AI Inference

OpenAI has released the first performance results from Jalapeño, its custom-designed inference chip, showing significant gains in both speed and energy efficiency compared with existing AI hardware.

According to OpenAI, Jalapeño is designed to handle modern AI workloads while reducing the traditional trade-off between high throughput and low latency. In practical terms, this could mean faster responses for users, more responsive AI agents, and greater capacity to serve growing demand without a proportional increase in energy consumption.

A New Approach to AI Inference

OpenAI evaluated Jalapeño using InferenceX, a public benchmark developed by SemiAnalysis that measures the performance of complete AI-serving systems rather than focusing solely on individual chips.

The company tested the architecture with three large open-weight models: GPT-OSS 120B, DeepSeek R1 670B, and Kimi K2.5 1T. Across these workloads, Jalapeño reportedly achieved between 1.5 and 1.9 times more AI work per watt at peak throughput, while end-to-end latency was reduced by between 1.7 and 3.6 times.

The results are particularly relevant for interactive AI applications, where even small delays can accumulate as an agent performs multiple steps. In these workloads, OpenAI says Jalapeño delivered between 2.1 and 4.1 times higher performance.

Designing the Entire System Around AI

Jalapeño was not developed simply as a faster processor. OpenAI designed the chip together with its memory, networking, software, and rack-scale infrastructure.

This approach addresses one of the central challenges of AI inference: different stages of model execution place different demands on hardware. During prefill, processing the user's prompt is primarily compute-intensive. During decoding, when the model generates a response token by token, memory bandwidth becomes increasingly important.

Communication between chips can also introduce delays. Jalapeño therefore emphasizes keeping model data close to the computing resources that need it, reducing unnecessary movement and communication.

The architecture is intended to provide a more balanced environment for both prefill and decoding, while remaining flexible enough to support different model architectures and increasingly agentic workloads.

AI Helped Build the Chip

One of the most notable aspects of the project is that AI itself played an important role in Jalapeño's development.

OpenAI says its teams used AI to explore hardware implementations, accelerate design and verification cycles, and optimize parts of the chip's arithmetic circuitry. The company moved from the initial design to tapeout in approximately nine months.

The chip was also designed to be relatively predictable from a programming perspective, making it easier for both human engineers and AI systems to optimize workloads across the architecture.

OpenAI reports that, using Codex and GPT-Astra, engineers were able to optimize three open-weight models that were not originally part of the chip's production plan within two months. For selected attention and mixture-of-experts components in GPT-OSS, AI-generated implementations were reportedly 1.5 to 1.8 times faster than existing implementations written by human experts.

These figures apply to specific components rather than complete models, but they illustrate a potentially important development cycle in which AI can help design hardware and subsequently optimize the software running on it.

Strong Results Across Different Models

Jalapeño showed competitive results across all three models tested.

For GPT-OSS 120B, OpenAI reports approximately 1.9 times higher peak throughput per watt than the comparison system, alongside roughly 1.7 times lower end-to-end latency.

On DeepSeek R1 670B, the chip achieved approximately 1.7 times higher peak performance per watt and 3.6 times lower end-to-end latency.

The results for Kimi K2.5 1T, the largest model included in the public testing, showed around 1.5 times higher peak performance per watt and 3.4 times lower end-to-end latency.

Together, these results place Jalapeño on what OpenAI describes as the Pareto frontier across the tested operating range, meaning it provides a particularly strong combination of performance and energy efficiency rather than optimizing only for one metric.

What Jalapeño Could Mean for AI Infrastructure

The importance of Jalapeño extends beyond benchmark numbers. As AI models become larger and AI agents perform increasingly complex tasks, infrastructure efficiency is becoming a central part of the economics of AI.

More useful computation from the same amount of power and hardware could allow AI providers to serve more users while controlling infrastructure costs. Lower latency could also enable new applications in which rapid, repeated model interactions are essential.

OpenAI says Jalapeño could make ultra-fast inference more efficient, improve the economics of fast inference, and increase the efficiency of batch workloads.

The Beginning of a Longer Hardware Roadmap

OpenAI plans to begin deploying Jalapeño within its own computing infrastructure by the end of 2026. The company describes the chip as the first generation of a broader, multigenerational platform, with a second generation already in development and a third generation taking shape.

At the same time, OpenAI says it will continue using accelerators from NVIDIA and other partners for both training and inference.

Jalapeño therefore represents not a replacement for the broader AI hardware ecosystem, but an additional layer of OpenAI's strategy to control and optimize more of the infrastructure behind its AI products.

The larger goal is clear: as demand for AI continues to grow, improving inference speed and energy efficiency will be essential. Jalapeño is OpenAI's attempt to address that challenge by designing hardware, software, networking, and AI models as parts of a single system.

The Role of a Product Manager Across the Product Lifecycle

A Product Manager (PM) plays a central role in turning ideas and customer problems into successful products. But the role of a PM is not limited to writing requirements, managing a roadmap, or coordinating a development team.

The Product Manager stays involved throughout the entire product lifecycle, while the focus of the role changes depending on where the product is in its journey.

From identifying the right problem to learning from a product after launch, the PM helps the team make better decisions, stay aligned, and ultimately build something valuable for users and the business.

Problem → Strategy → Planning → Design → Engineering → Testing → Launch → Learning

1. Identifying and Defining the Problem

One of the most important responsibilities of a Product Manager is making sure the team is solving the right problem.

Before thinking about features or solutions, PMs need to understand users, their needs, the market, and the competitive landscape. They may work with research, design, data science, and other teams to gather insights through:

  • User research

  • Market research

  • Competitive analysis

  • Product analytics

  • Customer feedback

  • Support insights

  • Experiments and hypothesis testing

The goal is not simply to find something that could be built, but to understand which problem is worth solving and why.

A strong PM spends significant time framing the problem clearly so that the entire team understands what they are trying to accomplish.

2. Developing the Product Strategy

Once the problem and opportunity are understood, the Product Manager helps define the strategy for solving it.

This means establishing a clear vision for the product and determining what it should accomplish. The PM connects product decisions to broader business objectives and defines the metrics that will indicate whether the product is successful.

A good product strategy answers questions such as:

  • Who are we building for?

  • What problem are we solving?

  • Why is this problem important?

  • What should the product accomplish?

  • How does it support the company's goals?

  • How will we measure success?

The PM does not necessarily decide every detail of the solution alone. Instead, they bring the right people together and create the context needed for the team to make effective decisions.

3. Planning the Work

After defining the strategy, the team needs to turn it into an executable plan.

Product Managers work closely with designers and engineers to understand the scope and complexity of the work. Large initiatives are broken down into smaller milestones, projects, and tasks.

Depending on the organization, PMs may collaborate with:

  • Product designers

  • iOS and Android engineers

  • Backend engineers

  • Data scientists

  • QA engineers

  • Technical or program managers

The PM's role is not to manage every task personally. Instead, they help establish priorities, clarify dependencies, identify risks, and make sure everyone understands what needs to happen and why.

4. Working Through UX and Product Design

Design is another critical stage of the product lifecycle.

Product Managers work closely with designers as they explore different ways to solve the user's problem. Designers may create wireframes, prototypes, mockups, and detailed UX specifications.

The PM contributes product context, customer insights, business requirements, and an understanding of the broader strategy.

Product intuition can be useful, but intuition is not always correct. When a design decision is particularly important or uncertain, teams can use usability testing or other forms of research to validate their assumptions.

For smaller changes, experimentation and A/B testing can sometimes provide a faster way to understand how users respond.

The key principle is simple: whenever possible, decisions should be informed by evidence rather than assumptions.

5. Supporting Engineering Implementation

Once the design and requirements are sufficiently clear, engineering begins implementation.

Before development starts, the Product Manager should make sure engineers understand what is being built and why. This is also the moment to ask a simple but important question:

"Is anything unclear or missing?"

During development, the PM continues to stay involved. They monitor progress, answer questions, help resolve trade-offs, and work to remove blockers.

The PM may also partner with a program or technical program manager to maintain visibility into timelines, dependencies, and risks.

The objective is not to tell engineers how to write the code. It is to make sure the team has the clarity and support necessary to build the right product effectively.

6. Testing and Validating the Product

Before launch, the product needs to be tested carefully.

Product Managers typically work with QA to review the testing strategy and ensure that important user scenarios and edge cases are covered.

PMs should also use the product themselves. Seeing the product firsthand can reveal issues that may not be obvious from requirements, designs, or project updates.

At this stage, the PM helps determine whether the product meets the expected quality bar and whether important bugs or usability problems need to be addressed before launch.

7. Launching the Product

Launching a product can look very different depending on its size, audience, and risk.

A small feature may be released quietly to a limited number of users. A major product launch might involve marketing campaigns, public relations, websites, blog posts, sales enablement, or even a launch event.

The Product Manager coordinates with the different teams involved and helps ensure that everyone is prepared.

For higher-risk products, a phased rollout can be especially valuable. Instead of releasing the product to everyone at once, the team can gradually increase exposure while monitoring product metrics, customer feedback, and potential issues.

This approach allows the team to react quickly if something unexpected happens.

8. Reviewing and Learning After Launch

The Product Manager's job does not end when the product launches.

In many ways, the launch is the beginning of the next stage of learning.

After launch, PMs analyze whether the product achieved its intended goals. They look at key metrics, customer feedback, user sentiment, and business impact.

They also ask:

  • Did users adopt the product?

  • Did we solve the original problem?

  • Did the key metrics improve?

  • What surprised us?

  • What went wrong?

  • What worked particularly well?

  • What should we change next time?

Successful teams treat launches as learning opportunities. They celebrate wins, acknowledge mistakes, and use what they learn to improve future products.

The Product Manager as a Cross-Functional Leader

Product Managers work with almost everyone involved in bringing a product to customers.

They may collaborate with:

TeamRole
DesignCreates the product experience and interface
ResearchProvides user, market, and product insights
EngineeringBuilds and maintains the product
TPM / Program ManagementHelps coordinate execution and timelines
QATests product quality and functionality
Data ScienceProvides analytics and experimentation insights
MarketingCommunicates the product to customers
PRCommunicates the product externally and to the media
SalesHelps sell the product
SupportHelps customers use the product
Legal & PrivacyHelps manage legal and privacy risks
PolicyDefines rules for how the product can be used
OperationsHelps deliver and operate the product
InternationalizationHelps adapt the product for different countries and languages

This cross-functional nature is one of the defining characteristics of product management.

PMs generally do not have direct authority over all these teams. Instead, they influence through clarity, communication, prioritization, and alignment.

Communication Is a Core PM Skill

One of the most important things a Product Manager does is communicate.

The best PMs make sure that everyone involved understands the same fundamental questions:

What are we building?

Why are we building it?

Who are we building it for?

How will we know if it is successful?

Product Requirements Documents (PRDs), presentations, meetings, written updates, and one-on-one conversations are all tools that PMs can use to create alignment.

A simple way to test whether communication is working is to ask five people on the team:

"What are we building and why?"

If everyone gives essentially the same answer, the team is probably aligned.

If five people give six different answers, the PM still has work to do.

PMs Must Respond to Change

Product development rarely goes exactly according to plan.

New customer research can challenge an existing assumption. An experiment can produce unexpected results. A competitor can launch a new product. Engineering can discover a technical constraint. Customer support can identify a major issue.

Sometimes there is an even more urgent problem, such as an outage or critical production bug.

Product Managers need to respond to this new information and continuously reassess priorities.

This requires the ability to juggle multiple initiatives, switch context quickly, and make decisions under uncertainty.

Being a PM is therefore not simply about following a plan. It is about continuously adapting the plan as new information becomes available.

The Product Manager as the Product's "Mini CEO"

Product Managers are sometimes described as "mini CEOs" of their products.

The analogy is useful because PMs are expected to take broad responsibility for the success of the product, from the initial idea through launch and beyond.

A PM may be expected to:

  • Identify and define problems

  • Create product strategy

  • Prioritize opportunities

  • Drive cross-functional alignment

  • Represent the product internally and externally

  • Coordinate development and launch

  • Respond to new information

  • Help remove blockers

  • Measure outcomes

  • Learn from failures and successes

However, being the "CEO" of a product does not mean having authority over everyone.

Instead, effective Product Managers lead through influence rather than hierarchy. They bring together people with different skills and perspectives and help the team make decisions in the best interest of the customer and the business.

The Big Picture

Product management is ultimately about helping a team figure out what to build, why to build it, and how to learn whether it worked.

The PM's involvement spans the entire product lifecycle:

Problem → Strategy → Planning → Design → Engineering → Testing → Launch → Learning

At each stage, the PM's focus changes.

Early in the lifecycle, the PM may spend most of their time understanding customers and defining the problem. During development, they may focus more on alignment, prioritization, and removing blockers. Around launch, coordination becomes critical. After launch, the focus shifts toward metrics, feedback, and learning.

And because PMs often manage multiple products or initiatives simultaneously, they may be working on several different stages of this lifecycle at the same time.

That is what makes product management challenging—and exciting.

A great Product Manager does not simply make sure that something gets built.

A great Product Manager helps the team build the right thing, for the right users, for the right reasons—and learns continuously along the way.

Training TensorFlow Object Detection Models with AWS SageMaker: Troubleshooting Common Errors

Training an object detection model with the TensorFlow Object Detection API and Amazon SageMaker can be a powerful way to combine deep learning with scalable cloud infrastructure. However, setting up the environment can sometimes be challenging, especially when working with older tutorials and newer versions of the AWS SDK.

In this post, I will describe the main issues I encountered while working on a TensorFlow 2 Object Detection project using the Waymo Open Dataset, AWS SageMaker, Docker, and Amazon ECR, together with the solutions that allowed the training workflow to work correctly.

Project Overview

The project uses the TensorFlow Object Detection API to train and evaluate different object detection architectures on the Waymo Open Dataset.

The dataset was already converted into TFRecord format and stored in Amazon S3. The images have a resolution of 640 × 640 pixels.

The main workflow is:

  1. Store the training and validation data in Amazon S3.

  2. Build a Docker container containing the TensorFlow Object Detection API.

  3. Push the Docker image to Amazon Elastic Container Registry (ECR).

  4. Download a pretrained model from the TensorFlow Model Zoo.

  5. Configure the training pipeline.

  6. Launch a SageMaker training job.

  7. Evaluate the model and compare different experiments.

The architectures considered include SSD MobileNet, SSD ResNet50, Faster R-CNN, EfficientDet, and Faster R-CNN ResNet152.

Problem 1: sagemaker.estimator Cannot Be Imported

The first error I encountered was:

ModuleNotFoundError: No module named 'sagemaker.estimator'

The problematic import was:

import sagemaker
from sagemaker.estimator import Estimator
from framework import CustomFramework

The reason was a compatibility issue between the notebook and the installed SageMaker Python SDK.

The original project was designed around SageMaker SDK version 2, while newer environments may install version 3 by default. The SageMaker SDK v3 introduced significant API changes, so older code relying on classes such as Estimator may no longer work.

Solution

Instead of installing the newest version of SageMaker, I installed a version from the 2.x series:

%pip install "sagemaker<3" tensorflow_io

After restarting the notebook kernel, the following imports worked again:

import sagemaker
from sagemaker.estimator import Estimator
from framework import CustomFramework

This produced a deprecation warning explaining that SageMaker SDK v2 is no longer the actively developed version.

For this particular project, however, using SDK v2 is appropriate because the provided CustomFramework implementation was designed around the older API.

Problem 2: SageMaker Could Not Find the Docker Image

After solving the SDK issue, the training job failed with another error:

An error occurred (ValidationException) when calling the CreateTrainingJob operation:

Cannot find the requested image:
166664655187.dkr.ecr.us-east-1.amazonaws.com/tf2-object-detection:20260817082045

At first, this looked like a SageMaker problem, but the important part of the message was the ECR image name and tag.

SageMaker was trying to download an image from Amazon ECR, but that exact image did not exist.

The project uses a script similar to:

./docker/build_and_push.sh tf2-object-detection

to build the Docker image and push it to ECR.

I checked the repository with:

aws ecr describe-images \
    --repository-name tf2-object-detection \
    --region us-east-1

This helped determine whether the timestamped image tag actually existed.

Problem 3: Docker Was Not Authorized to Push to ECR

The final error revealed the real cause:

denied: User:
arn:aws:sts::166664655187:assumed-role/AmazonSageMaker-ExecutionRole-20260817T095324/SageMaker
is not authorized to perform:
ecr:InitiateLayerUpload

This was the key discovery.

The SageMaker execution role did not have enough permissions to upload Docker image layers to Amazon ECR.

The build script was therefore unable to push the image successfully. However, it still wrote an image URI into:

docker/ecr_image_fullname.txt

This explained the previous error: SageMaker received an image URI, but the corresponding image had never actually been uploaded to ECR.

Granting ECR Permissions

The SageMaker execution role needs permissions that allow it to interact with ECR.

For a development or educational project, one straightforward solution is to attach the AWS-managed:

AmazonEC2ContainerRegistryPowerUser

policy to the SageMaker execution role.

The relevant role in my environment was:

AmazonSageMaker-ExecutionRole-20260817T095324

In the AWS IAM console, the process is:

  1. Open IAM.

  2. Open Roles.

  3. Select the SageMaker execution role.

  4. Choose Add permissions.

  5. Select Attach policies.

  6. Search for AmazonEC2ContainerRegistryPowerUser.

  7. Attach the policy.

For production environments, it is generally better to use a more restrictive custom IAM policy that grants only the ECR actions and repository resources actually required.

Rebuilding and Pushing the Image

After updating the IAM permissions, I ran the Docker build and push script again:

./docker/build_and_push.sh tf2-object-detection

This time the Docker image could be uploaded to ECR successfully.

I then verified the repository:

aws ecr describe-images \
    --repository-name tf2-object-detection \
    --region us-east-1

The newly generated timestamp tag was now visible.

I also refreshed the container variable:

with open('docker/ecr_image_fullname.txt', 'r') as f:
    container = f.read().strip()

print(container)

The value looked similar to:

166664655187.dkr.ecr.us-east-1.amazonaws.com/tf2-object-detection:20260817103000

The important point is that this exact tag must exist in ECR before starting the SageMaker training job.

Launching the Training Job

Once the Docker image was available in ECR, the SageMaker estimator could be created:

estimator = CustomFramework(
    role=role,
    image_uri=container,
    entry_point='run_training.sh',
    source_dir='source_dir/',
    hyperparameters={
        "model_dir": "/opt/training",
        "pipeline_config_path": "pipeline.config",
        "num_train_steps": "2000",
        "sample_1_of_n_eval_examples": "1"
    },
    instance_count=1,
    instance_type='ml.g5.xlarge',
    tensorboard_output_config=tensorboard_output_config,
    disable_profiler=True,
    base_job_name='tf2-object-detection'
)

The training job can then be started with:

estimator.fit(inputs)

SageMaker launches the specified GPU instance, downloads the Docker image from ECR, accesses the training data from S3, and executes the training script inside the container.

Lessons Learned

This project highlighted several important lessons about machine learning workflows in the cloud.

1. Check software versions

Older machine learning tutorials can depend on APIs that have changed significantly.

Before troubleshooting the code itself, it is useful to check:

import sagemaker
print(sagemaker.__version__)

Using the SDK version expected by the project can save considerable debugging time.

2. An image URI does not guarantee that an image exists

A file such as:

ecr_image_fullname.txt

may contain a valid-looking ECR URI even if the Docker push failed.

Always verify the image directly:

aws ecr describe-images \
    --repository-name tf2-object-detection \
    --region us-east-1

3. IAM permissions are critical

AWS services frequently interact through IAM roles. A SageMaker execution role may have permission to run training jobs but still lack permission to push or pull Docker images from ECR.

The specific error message is often very useful because it identifies the missing action, such as:

ecr:InitiateLayerUpload

4. Verify the complete pipeline before starting an expensive training job

GPU instances such as ml.g5.xlarge can be expensive. It is better to verify the following before launching training:

  • The S3 training data is accessible.

  • The S3 validation data is accessible.

  • The Docker image builds correctly.

  • The Docker image is successfully pushed to ECR.

  • The ECR image tag exists.

  • The SageMaker execution role has the required permissions.

  • The pipeline.config file matches the selected model architecture.

This avoids wasting compute time on infrastructure problems.

Next Steps: Improving the Object Detection Model

Once the infrastructure is working, the next challenge is improving model performance.

The initial experiment can be used as a baseline. Further experiments can focus on:

  • Data augmentation.

  • Learning rate and optimizer configuration.

  • Number of training steps.

  • Batch size.

  • Different object detection architectures.

  • Different pretrained checkpoints.

  • Evaluation frequency.

  • Model-specific hyperparameters.

For example, TensorFlow's Object Detection API provides several augmentation techniques that can improve robustness to variations in lighting, scale, orientation, and image composition.

The experiments should be evaluated using metrics such as mean Average Precision (mAP), while TensorBoard can be used to analyze training and validation behavior.


The biggest challenge in this project was not necessarily the object detection model itself, but making the different components of the cloud training pipeline work together.

The final workflow can be summarized as:

Waymo Dataset
      ↓
Amazon S3
      ↓
TensorFlow Object Detection API
      ↓
Docker Container
      ↓
Amazon ECR
      ↓
Amazon SageMaker
      ↓
GPU Training
      ↓
Evaluation + TensorBoard

The two most important troubleshooting steps were using a compatible SageMaker SDK version and ensuring that the SageMaker execution role had the necessary ECR permissions.

Once these infrastructure issues were resolved, SageMaker could successfully use the custom TensorFlow Object Detection container and proceed with model training.

This experience also reinforced an important principle for cloud-based machine learning: before optimizing the model, make sure the entire data, container, permissions, and training pipeline is working reliably.

Understanding the AutoML Workflow: From Raw Data to Production

Automated Machine Learning (AutoML) is often described as a way to automate model selection and hyperparameter tuning. In practice, modern AutoML systems can automate a much broader machine learning workflow, from preparing datasets to evaluating models and generating production-ready pipelines.

For software developers, understanding this workflow is important because AutoML is not simply a "train button." It is an orchestration layer that combines data processing, feature engineering, model selection, hyperparameter optimization, validation, and deployment.

This article walks through the typical AutoML workflow and explains what happens at each stage.

1. Data Ingestion and Validation

Every AutoML workflow starts with data.

The system first loads the training dataset and determines its structure. Depending on the framework, this may involve detecting:

  • Numerical and categorical columns

  • Missing values

  • Text or timestamp fields

  • The target variable

  • Class distributions

  • Potentially irrelevant or duplicated features

Data validation is particularly important because machine learning algorithms generally expect structured and consistent input.

A typical pipeline might conceptually look like:

Raw Dataset
     |
     v
Schema Detection
     |
     v
Data Validation
     |
     v
Preprocessing

At this stage, an AutoML system may identify problems such as missing values, inconsistent data types, highly imbalanced classes, or columns that should not be used as predictors.

However, developers should not assume that automated validation can detect every data-quality problem. Business-specific constraints often require explicit validation rules.

2. Data Preprocessing

Once the dataset has been validated, AutoML applies preprocessing transformations.

Common operations include:

  • Missing-value imputation

  • Numerical feature scaling

  • Categorical encoding

  • Outlier handling

  • Feature normalization

  • Text vectorization

  • Date and time transformations

For example, a categorical feature such as:

country = ["US", "UK", "DE", "FR"]

may be transformed using one-hot encoding or another representation.

A key requirement is that preprocessing must be reproducible. The transformations applied during training must also be applied consistently during inference.

A robust AutoML pipeline therefore treats preprocessing as part of the model pipeline rather than as a separate manual operation.

3. Feature Engineering

Feature engineering is one of the most valuable—and potentially most complex—parts of AutoML.

Traditional machine learning workflows often rely heavily on domain experts to create useful features. AutoML systems attempt to automate some of this process.

For example, a timestamp could generate features such as:

year
month
day_of_week
hour
is_weekend

A transaction dataset could potentially generate aggregate features such as:

average_transaction_value
transactions_last_30_days
customer_lifetime_value

Some AutoML systems also perform feature selection, removing features that provide little predictive value.

Feature engineering can significantly improve model performance, but automated transformations must be designed carefully to avoid data leakage.

4. Train/Test Splitting and Cross-Validation

The next step is determining how models will be evaluated.

For a typical supervised learning problem, the dataset is divided into training and validation data, with a separate test set reserved for final evaluation.

A simplified structure is:

Dataset
   |
   +---- Training Set
   |
   +---- Validation Set
   |
   +---- Test Set

AutoML systems may use cross-validation instead of a single validation split.

For example, with five-fold cross-validation, the training data is divided into five subsets. The model is trained multiple times, using different subsets for validation.

This provides a more reliable estimate of generalization performance, particularly when the dataset is relatively small.

Developers should pay close attention to the splitting strategy. Random splitting is not appropriate for every problem. Time-series data, for example, generally requires time-aware validation to prevent future information from leaking into the training process.

5. Model Selection

Once the data pipeline is established, the AutoML system can evaluate different machine learning algorithms.

Depending on the task, candidates might include:

Classification

  • Logistic Regression

  • Decision Trees

  • Random Forests

  • Gradient Boosting

  • Support Vector Machines

  • Neural Networks

Regression

  • Linear Regression

  • Random Forest Regression

  • Gradient Boosting

  • Neural Networks

Other Tasks

Specialized AutoML systems may also support:

  • Time-series forecasting

  • Natural language processing

  • Computer vision

  • Recommendation systems

Rather than manually selecting one algorithm, AutoML evaluates multiple candidates under a common evaluation framework.

Conceptually:

                    +--> Model A --> Score
                    |
Training Pipeline --+--> Model B --> Score
                    |
                    +--> Model C --> Score

The goal is not necessarily to find the theoretically "best" algorithm. Instead, the system searches for a model configuration that performs well under the selected constraints and evaluation metric.

6. Hyperparameter Optimization

Selecting an algorithm is only part of the problem.

Most machine learning algorithms have hyperparameters that influence their behavior.

For example, a gradient-boosting model may have parameters controlling:

learning_rate
number_of_trees
maximum_depth
subsample_ratio

An AutoML system can automatically search through different combinations of these values.

Common optimization strategies include:

  • Grid search

  • Random search

  • Bayesian optimization

  • Evolutionary optimization

  • Hyperband and other resource-aware strategies

Modern AutoML systems often combine model selection with hyperparameter optimization rather than treating them as completely independent steps.

The search can therefore be represented as:

Algorithm
    +
Hyperparameters
    +
Preprocessing Choices
    |
    v
Candidate Pipeline
    |
    v
Evaluation

The system repeats this process until it reaches a specified time, compute, or experiment budget.

7. Evaluation and Optimization Metrics

AutoML needs an objective function to determine which candidate is better.

The metric depends on the business and technical requirements.

For classification, common metrics include:

  • Accuracy

  • Precision

  • Recall

  • F1 score

  • ROC-AUC

  • Log loss

For regression:

  • Mean Absolute Error (MAE)

  • Mean Squared Error (MSE)

  • Root Mean Squared Error (RMSE)

Choosing the correct metric is critical.

For example, consider a fraud-detection system where fraudulent transactions represent only a small percentage of all transactions. Optimizing for accuracy alone could produce a model that appears highly accurate while detecting very few fraudulent transactions.

The AutoML system can optimize only what it is instructed to optimize. Therefore, metric selection remains a developer and domain-expert responsibility.

8. Experiment Tracking

A serious AutoML workflow can generate hundreds or thousands of candidate experiments.

Each experiment may contain:

Dataset version
Preprocessing configuration
Algorithm
Hyperparameters
Validation strategy
Evaluation metrics
Training duration
Model artifacts

Experiment tracking becomes essential for reproducibility.

Instead of simply keeping the best model, production systems should maintain information about how that model was produced.

This allows developers to answer questions such as:

  • Which dataset was used?

  • Which features were selected?

  • Which hyperparameters produced the model?

  • Which validation strategy was used?

  • Why was this model selected?

  • Can the experiment be reproduced?

AutoML therefore fits naturally into modern MLOps architectures.

9. Selecting the Final Model

After the search process finishes, AutoML ranks the candidate pipelines according to the optimization objective.

However, the model with the highest validation score is not automatically the best production model.

Developers may also need to consider:

  • Inference latency

  • Memory consumption

  • Model size

  • Interpretability

  • Infrastructure costs

  • Fairness requirements

  • Robustness

  • Security

  • Operational complexity

For example, a slightly less accurate model may be preferable if it has significantly lower inference latency and is easier to deploy.

This introduces an important principle:

AutoML optimizes the objective you define, not necessarily the system you actually need.

10. Final Training

Once the best configuration has been identified, the final model can be trained using the appropriate training data.

Depending on the workflow, the final training stage may use more data than the individual experiments.

The resulting artifact should include not only the trained model but also the preprocessing pipeline required to transform production data into the expected model input.

Conceptually:

Raw Input
   |
   v
Preprocessing
   |
   v
Feature Transformation
   |
   v
Trained Model
   |
   v
Prediction

Keeping these components together reduces the risk of training-serving skew.

11. Deployment

After validation, the model can be deployed.

A common architecture exposes the model through an API:

Application
     |
     | HTTP/gRPC
     v
Inference Service
     |
     v
Preprocessing
     |
     v
Model
     |
     v
Prediction

Depending on the requirements, the model may run:

  • As a REST API

  • Inside a container

  • As a batch-processing job

  • On a cloud ML platform

  • At the edge

  • Directly inside an application

The deployment strategy should take latency, scalability, availability, and cost into account.

12. Monitoring in Production

Deployment is not the end of the AutoML workflow.

A model can perform well during training and gradually become less effective as production data changes.

Monitoring should therefore cover several dimensions.

System Metrics

Examples include:

  • CPU and memory usage

  • Request latency

  • Throughput

  • Error rates

  • Availability

Data Metrics

Examples include:

  • Missing-value rates

  • Feature distributions

  • Input ranges

  • Category frequencies

Model Metrics

When ground-truth labels become available, developers can monitor:

  • Accuracy

  • Precision

  • Recall

  • Error rates

  • Business-specific KPIs

This makes it possible to detect data drift and model degradation.

13. Retraining and Continuous Optimization

When the production environment changes, the model may need to be retrained.

A mature AutoML architecture can automate parts of this process:

Production Data
      |
      v
Monitoring
      |
      v
Drift Detection
      |
      v
Retraining Trigger
      |
      v
AutoML Search
      |
      v
Model Validation
      |
      v
Deployment

However, fully automated deployment should be approached carefully.

In many production environments, a better approach is to automatically train and evaluate new candidates while requiring an approval step before replacing the production model.

This creates a controlled continuous-learning workflow.

AutoML as a Software Engineering Problem

For software developers, the most useful way to think about AutoML is not as a replacement for machine learning expertise but as an automation framework.

The overall pipeline can be summarized as:

Data
 |
 v
Validation
 |
 v
Preprocessing
 |
 v
Feature Engineering
 |
 v
Model Search
 |
 v
Hyperparameter Optimization
 |
 v
Cross-Validation
 |
 v
Model Selection
 |
 v
Final Training
 |
 v
Deployment
 |
 v
Monitoring
 |
 +----> Retraining

Each stage introduces engineering decisions that cannot always be automated safely.

AutoML can reduce the amount of manual experimentation required, but developers still need to define the problem correctly, establish reliable data pipelines, select appropriate metrics, control computational resources, and integrate models into production systems.

AutoML automates a large portion of the machine learning experimentation lifecycle, transforming what could be a highly manual process into a repeatable pipeline.

For software developers, its real value lies in the ability to systematically explore different combinations of preprocessing techniques, algorithms, features, and hyperparameters while maintaining an experiment-driven workflow.

The most effective AutoML implementations therefore combine automation with engineering discipline.


Server Monitoring with htop

When you are developing and maintaining applications on Linux servers, understanding what is happening at the system level can save you a lot of time.

An application may suddenly become slow, a deployment may consume more resources than expected, or a background process may start using an unusual amount of CPU or memory. Before reaching for complex monitoring platforms, there is a simple tool that can provide an immediate overview of the system: htop.

htop is an interactive process viewer for Unix-like systems. It provides a real-time view of running processes and system resource usage, making it particularly useful when troubleshooting servers from the command line.

What Is htop?

htop is an interactive system-monitoring tool that can be used to inspect processes and understand how a Linux or Unix-like system is using its resources.

It is often considered a more user-friendly alternative to the traditional top command. Instead of displaying a mostly text-based list of processes, htop provides a more interactive interface with visual resource indicators and keyboard shortcuts.

A typical htop screen gives you information about:

  • CPU utilization

  • Memory usage

  • Swap usage

  • System load

  • Running processes

  • Process IDs (PIDs)

  • CPU and memory consumption per process

  • Process ownership

  • Process priority and scheduling information

For developers working directly on servers, this information can be extremely valuable.

Installing htop

On Debian- and Ubuntu-based systems, you can usually install htop with:

sudo apt install htop

On Fedora:

sudo dnf install htop

On Arch Linux:

sudo pacman -S htop

Once installed, start it with:

htop

Because htop is interactive, you can navigate through processes using the keyboard rather than repeatedly running commands.

Understanding the htop Interface

One of the main advantages of htop is that important information is visible immediately.

At the top of the interface, you will typically find resource meters for CPU, memory, and swap, followed by system information such as load average and uptime.

The process list occupies most of the screen.

A process may be displayed with information such as:

  • PID

  • User

  • Priority

  • Nice value

  • Virtual memory

  • Resident memory

  • CPU percentage

  • Memory percentage

  • Execution time

  • Command

This makes it possible to quickly identify processes that are consuming an unusual amount of resources.

Monitoring CPU Usage

One of the most common reasons to use htop is investigating high CPU usage.

Suppose an API server suddenly becomes slow. You connect to the machine and run:

htop

The CPU meters can immediately tell you whether the machine is under heavy CPU load.

You can then sort the process list by CPU consumption to identify the processes responsible for the load.

This is particularly useful when dealing with:

  • CPU-intensive application code

  • Background workers

  • Build processes

  • Compilers

  • Database workloads

  • Containers

  • Unexpected processes

A high CPU percentage does not automatically mean there is a problem. A process may legitimately need significant CPU resources. The important question is whether the usage is expected and whether it correlates with the performance problem you are investigating.

Investigating Memory Usage

Memory problems are another common source of server instability.

In htop, the memory meter provides an immediate overview of RAM usage, while the process list helps you identify which processes are consuming the most memory.

For example, you might discover that a particular application process is using several gigabytes of RAM.

That could lead to further investigation:

  1. Is the application expected to use that much memory?

  2. Has memory usage increased over time?

  3. Are multiple instances running?

  4. Is the process leaking memory?

  5. Is the server running out of available memory?

  6. Is swap being used heavily?

htop does not diagnose the root cause of a memory leak, but it is an excellent first step for identifying suspicious processes.

Using htop During Production Incidents

One of the biggest strengths of htop is its simplicity.

During an incident, you may not have time to configure a complete monitoring stack. If you can access the server through SSH, you can often launch htop within seconds.

For example:

ssh user@server
htop

You can then quickly determine whether the problem appears to be related to CPU, memory, or a specific process.

This can help answer an important first question:

Is the application slow because of the application itself, or because the server is under resource pressure?

That distinction can significantly change the debugging strategy.

Finding a Specific Process

On a busy server, the process list can contain hundreds of entries.

Instead of manually searching through them, htop provides interactive navigation and filtering capabilities.

You can use the search functionality to find processes by name or command.

For example, if you are investigating a Node.js application, you can search for processes related to Node:

node

Similarly, developers working with Python, Java, PHP, Go, or other runtimes can quickly narrow the process list to relevant applications.

This is much faster than manually inspecting the output of a large process listing.

Sorting Processes

Sorting is one of the most useful features when troubleshooting resource usage.

If CPU usage is the problem, sort processes by CPU consumption.

If memory usage is the problem, sort by memory consumption.

This immediately puts the most resource-intensive processes at the top of the list.

Instead of asking:

"What is consuming all the memory?"

you can often answer the question within seconds.

Managing Processes

htop is not only a monitoring tool. It also provides controls for interacting with processes.

Depending on your permissions, you can perform actions such as:

  • Sending signals to processes

  • Terminating processes

  • Changing process priority

  • Searching for processes

  • Filtering the process list

  • Viewing process details

For example, if a process has become completely unresponsive, you may be able to select it and send an appropriate signal.

However, process management should be used carefully on production systems. Killing the wrong process can cause service interruptions or data loss.

Monitoring should come before intervention.

Understanding Load Average

The load average displayed by htop is another useful indicator, but it is important to interpret it correctly.

Load average represents the number of tasks that are either running or waiting for system resources, depending on the operating system's accounting.

For example, a server with many CPU cores can naturally have a higher load average than a single-core system without necessarily being overloaded.

Therefore, avoid interpreting load average as a simple percentage.

Instead, compare it with the number of available CPU cores and other indicators such as CPU utilization, I/O wait, and application behavior.

htop and Containers

Modern applications frequently run inside containers.

Although htop operates at the host level, it can still be useful when investigating containerized workloads because container processes ultimately consume host resources.

If a Docker host is experiencing high CPU or memory usage, htop can help you identify which processes are responsible.

For container-specific investigation, however, you may also want to use tools such as:

docker stats

or the monitoring tools provided by your container orchestration platform.

The best approach is often to combine application-level, container-level, and host-level information.

htop vs. top

Linux already includes top on most systems, so why use htop?

The main difference is usability.

top is lightweight and widely available, making it an excellent tool for minimal environments and recovery situations.

htop, on the other hand, provides a more interactive experience. It makes it easier to navigate processes, sort information, search, and understand resource consumption visually.

For developers who regularly work on Linux servers, htop can therefore be a more convenient day-to-day troubleshooting tool.

That does not make top obsolete. Knowing both is useful, especially when working with minimal systems where htop may not be installed.

A Practical Troubleshooting Workflow

A simple workflow can make htop particularly effective.

When a server starts behaving unexpectedly:

1. Connect to the server

ssh user@server

2. Start htop

htop

3. Check CPU usage

Look for consistently high CPU utilization and identify the processes responsible.

4. Check memory

Look at RAM and swap usage and determine whether a specific process is consuming an unusual amount of memory.

5. Inspect suspicious processes

Check the process name, user, command, and resource consumption.

6. Correlate the information

Compare what you see with application logs, deployment activity, database metrics, and other monitoring systems.

7. Take action carefully

Only after identifying the likely cause should you consider restarting services, terminating processes, scaling resources, or making configuration changes.

This workflow turns htop into a fast first-response tool rather than simply a process viewer.

What htop Cannot Tell You

Although htop is extremely useful, it should not be treated as a complete observability platform.

It tells you a lot about what the operating system is doing, but not necessarily why your application is doing it.

For example, htop might show that your application is consuming 100% of a CPU core. It will not tell you which function or request is responsible.

For deeper investigations, you may need:

  • Application logs

  • Metrics

  • Distributed tracing

  • Database monitoring

  • Profiling tools

  • APM platforms

  • Container and orchestration metrics

The best production troubleshooting strategy combines these different levels of visibility.