Think of Docker as a consistent application box. The box contains the application files, runtime, dependencies, default command, and environment expectations. When another developer runs the same image, they get the same packaged environment.

Docker does not remove the need for testing, security, database design, monitoring, or deployment planning. It only makes the application package more repeatable. That repeatability is very powerful in development, QA, CI/CD, and production-like deployments.

For new learners, always connect Docker to one idea: consistency. If you understand consistency, image, container, Dockerfile, registry, and Compose become easier.

This architecture matters because Docker errors can come from different places. If the Docker CLI is installed but the daemon is not running, commands fail. If the daemon is running but cannot reach the registry, image pulls fail. If an image exists but the application command is wrong, the container exits. Good troubleshooting starts by locating the failing layer.

Docker Desktop on Windows/macOS hides some complexity, but the same idea remains. Docker Engine runs the daemon. The daemon uses the container runtime and operating system features to start containers. The Docker root directory stores images, containers, volumes, and build cache.

Images are read-only templates. They contain layers, files, runtime dependencies, environment defaults, and startup metadata. Containers are created from images. When a container runs, Docker adds a writable layer on top of the image layers.

A registry stores images so other machines can pull them. Docker Hub is a public registry. Companies also use private registries such as Amazon ECR, GitHub Container Registry, Azure Container Registry, or internal registries.

If you remove a container, the image may still remain. If you remove an image, existing containers may still exist but you cannot create new containers from that image unless it is rebuilt or pulled again.

docker run creates and starts a new container. docker ps shows running containers. docker ps -a shows stopped and running containers. docker stop stops a running container. docker start starts an existing stopped container. docker rm removes a stopped container. docker logs shows what the container process printed.

This lifecycle is different from a virtual machine. A container is not meant to be a full computer that keeps running forever by default. It runs the command configured for the image or passed at runtime.

This chapter is essential because many Docker beginner problems are port problems. A container can be running perfectly, but if the port is not published, your browser cannot access it from the host. Also, EXPOSE in a Dockerfile is only documentation. It does not publish the port. Publishing happens with docker run -p, Compose ports, or orchestration platform settings.

In production, users may not directly access Docker host ports. Traffic may come through load balancers, reverse proxies, ingress controllers, or service meshes. But the concept remains: traffic needs a defined route to the container service.

A Dockerfile is part of your project source code. It should be versioned with the application. If a new developer clones the repository, the Dockerfile tells them how to build the application image.

A good Dockerfile is not only working; it is maintainable. It should be readable, cache-friendly, secure, and not include unnecessary files or secrets. A Dockerfile affects build speed, image size, vulnerability count, and production reliability.

Start simple, then improve. First build a working image. Then add .dockerignore. Then improve cache. Then run as non-root. Then reduce image size. Then scan vulnerabilities.

FROM chooses the base image. RUN executes commands while building the image. COPY copies files from your project into the image. WORKDIR sets the directory for later commands. ENV sets environment variables available in the container. ARG sets build-time variables. EXPOSE documents the intended port. USER changes the user that runs the process. CMD gives a default command. ENTRYPOINT sets the fixed executable.

CMD vs ENTRYPOINT is confusing. Use CMD for simple default startup commands. Use ENTRYPOINT when the image should behave like a command-line tool or fixed executable. CMD can provide default arguments to ENTRYPOINT.

Each instruction can affect cache, security, image size, and runtime behavior.

This matters for speed and security. If node_modules, venv, .git, logs, screenshots, build output, or .env files enter the context, builds become slower and riskier. Even if a secret is not copied into the final image, sending it into a build environment is bad practice.

A good .dockerignore keeps the context clean. It also helps build cache because unnecessary file changes do not invalidate build steps. Teams should treat .dockerignore as a normal required project file, not an optional extra.

Students should always create .dockerignore when creating a Dockerfile.

Docker reads Dockerfile instructions in order. If an instruction changes, that step and following steps may rebuild. So stable steps should come before frequently changing steps.

For Python, copy requirements.txt first and install dependencies. Then copy the source code. For Node.js, copy package.json/package-lock.json first, run npm ci, then copy the rest. This way, code changes do not reinstall dependencies unless dependency files changed.

Cache optimization is a major business benefit in CI/CD because it reduces build time and compute cost.

Layers come from image history and build steps. If two images use the same base layer, Docker can share that layer. This is why image pulling and caching can be efficient.

Tags are convenient but can move. For example, latest may point to one image today and another image tomorrow. For production, this can be risky if you do not know exactly what was deployed. Digests are content-addressed and identify the exact image.

A serious release process often records image name, tag, and digest. This helps rollback and audit.

The first stage is usually the build stage. It may contain compilers, package managers, source code, and dev dependencies. The final stage is the runtime stage. It should contain only what is needed to run the app.

This makes images smaller and safer. Smaller images pull faster, deploy faster, and often contain fewer vulnerable packages. Multi-stage builds are common in professional Dockerfiles.

Do not use multi-stage blindly. Always test the final image because sometimes required runtime files are accidentally not copied.

BuildKit improves build performance and capabilities. It can skip unused stages, build independent stages in parallel, transfer only changed files, and support advanced features like build secrets.

buildx is commonly used for multi-platform builds. This matters because your laptop may be ARM64 and your production server may be AMD64. A multi-platform image can support both.

BuildKit also supports safer build secrets. Instead of using ARG or ENV for secret values, secret mounts make sensitive data available only during the build step that needs it.

A good image contains application code and dependencies. Environment-specific values should be provided at runtime. Common runtime values include database host, log level, API URL, feature flag, and environment name.

Do not store secrets in the Dockerfile. For local development, .env files are common, but do not commit real secrets. In production, use a secret manager or platform secret system.

Runtime settings also include memory limits, CPU limits, restart policies, container name, network, and command overrides. These settings define how the container behaves after image build.

Docker images are read-only. A running container gets a writable layer on top. This writable layer stores changes made during runtime. It is useful for temporary files, but not for important business data.

If a database writes data only into the container writable layer, removing the container can remove the data. This is why Docker provides volumes, bind mounts, and other storage options.

The core rule for students: container can be temporary; important data needs persistent storage.

Volumes are managed by Docker and stored in Docker's storage area. They are preferred for data generated and used by containers because they are independent of one specific container lifecycle.

When a database container uses a volume, the database writes to a path inside the container, but that path maps to the volume. If the container is removed and recreated with the same volume, the data can still exist.

Remember: volume persistence is not the same as backup. A volume can still be deleted or corrupted. Important data needs backup and restore planning.

Bind mounts are common in development because they let source code on the host be used by the container. You can edit code in VS Code and run it inside a container without rebuilding every time.

But bind mounts depend on host paths and permissions. If the host path does not exist, or if permissions differ, the container may fail. Bind mounts can also hide files already inside the image at the mount destination. This can surprise students.

For persistent container-managed data, volumes are usually better. For local code editing, bind mounts are very useful.

For real business systems, persistence and backup are different. Persistence means data survives normal container recreation. Backup means data can be recovered after deletion, corruption, or system failure.

For databases, official database dump tools are often safer than raw file copies. For learning, you can use a temporary container to tar volume contents. The most important lesson is: test restore. A backup you cannot restore is not useful.

Document backup frequency, retention, encryption, owner, and restore procedure for business systems.

Docker uses network drivers. The bridge driver is common for local containers. Containers on a user-defined bridge network can communicate and resolve each other by name. This name-based communication is very important in Compose.

There are two separate ideas: container-to-container communication and host-to-container access. Container-to-container communication uses Docker networks. Host-to-container access usually requires port publishing.

Do not publish internal services unnecessarily. If only the API needs to talk to the database, the database does not need a host port.

For host-to-container access, use -p host_port:container_port. For container-to-container access, use Docker network service names. In Compose, if your database service is named db, the app should connect to db:5432.

Also check application bind address. Many frameworks bind to 127.0.0.1 by default. Inside a container, this may prevent traffic from reaching the app through published ports. For containers, apps usually need to listen on 0.0.0.0.

Understanding localhost, port publishing, and service names solves many Docker issues.

Troubleshooting should be layered. For host access, check docker ps, docker port, curl from host, and application logs. For container-to-container access, exec into one container and test DNS or connection to the other service name.

If DNS fails, check whether both containers are on the same user-defined network. If DNS works but connection fails, check the service port and application readiness. If external internet fails, check DNS, proxy, VPN, Docker daemon network, and firewall.

Write results down. Network issues are easier when you compare expected path and actual failure point.

Compose describes services, networks, volumes, ports, environment variables, dependencies, builds, images, commands, and health checks. When you run docker compose up, Compose creates the needed network and starts services.

Compose is excellent for local development, QA, demos, and learning. It teaches how real applications are made of multiple components. In Compose, services can communicate by service name. If the database service is named db, the API can use DB_HOST=db.

Compose is not only a shortcut. It is a readable description of your application architecture.

A service can use an existing image or build from a Dockerfile. It can publish ports, define environment variables, mount volumes, define dependencies, and run health checks.

Compose creates a default network, but you can define custom networks. Services on the same network can reach each other by service name. You can also define named volumes for databases.

For students, the goal is to look at compose.yaml and understand the full app stack: what starts, what talks to what, what is public, and what data persists.

depends_on can control order, but a service being started does not always mean it is ready. Compose supports health checks and dependency conditions such as service_healthy. The database can be marked healthy only after a command like pg_isready succeeds.

Applications should also have retry logic. Even if the database is healthy at startup, it can restart later. A robust app should retry connections instead of crashing permanently.

Health checks are also important in production platforms. They help decide if traffic should be sent to a service.

Profiles keep Compose files clean. Instead of maintaining separate compose files for every optional tool, you can mark services with profiles. Then start them only with --profile.

Compose Watch improves development speed. It can sync source files into running containers, rebuild when dependency files change, or restart services. It is useful when you want a fast edit-test loop.

Students should still understand Dockerfile, volumes, and bind mounts. Watch is not magic. It works best when you know which files should sync and which changes require rebuild.

Public images are commonly pulled from Docker Hub. Companies often use private registries to protect internal application images. You usually need docker login before pushing to a private registry.

Tagging strategy is important. latest is convenient but risky for serious deployments. Use version tags, Git SHA tags, or build numbers. For production, record the digest too.

A good release flow builds once, tests, scans, pushes, and promotes the same image across environments.

A practical pipeline may do these steps: checkout code, build image, run unit tests, run integration tests, scan image, tag with Git SHA, push to registry, deploy to test environment, run smoke test, and promote to production after approval.

The image should be traceable to source code. If production fails, you should know exactly which Git commit and image tag/digest is running. This is important for rollback, audit, and incident response.

Docker helps ensure tests run against the same runtime package that will be deployed.

Security starts during build. Use trusted base images, keep dependencies updated, avoid unnecessary packages, use .dockerignore, and scan images. Do not copy .env files, SSH keys, tokens, or passwords into images.

Runtime security means least privilege. Run as non-root when possible. Avoid privileged containers. Drop unnecessary Linux capabilities. Publish only required ports. Avoid mounting the Docker socket unless absolutely necessary.

Operational security includes registry permissions, vulnerability policies, logging, monitoring, and incident response.

Build-time secrets and runtime secrets are different. If a build needs private package access, use BuildKit secret mounts instead of ARG or ENV. Build arguments and environment variables can persist in metadata or image layers depending on use, so they are not appropriate for sensitive build secrets.

For runtime, production systems should use secret managers or orchestrator secret features. For local practice, .env is common, but real secrets must not be committed to Git or copied into images.

The safe principle is: an image should be shareable without exposing credentials.

SBOM means Software Bill of Materials. It is like an ingredient list for software. Docker Scout can analyze images and show known vulnerabilities. This helps teams find risky packages before production.

A scan result is not the final answer by itself. Teams must review severity, exploitability, base image updates, package updates, and exceptions. Some vulnerabilities are fixed by updating base images. Some require application dependency updates.

Scanning should happen in CI/CD and after release too, because new vulnerabilities can be discovered after an image was built.

Start by classifying the problem. Is it build failure, image pull failure, container exit, app unreachable, database connection issue, permission issue, volume issue, registry issue, or security scan issue?

For runtime issues, check docker ps -a, logs, exit code, inspect, port mapping, environment variables, networks, and volumes. For build issues, check Dockerfile step, build context, .dockerignore, package install logs, and internet/proxy. For registry issues, check login, image name, tag, permissions, and platform.

A professional troubleshooting note includes symptom, scope, command, expected result, actual result, root cause, fix, and prevention.

Use --progress=plain to see detailed output. If COPY fails, check build context, file path, and .dockerignore. If RUN pip install or npm ci fails, read dependency errors. If apt-get fails, check package name and network access. If command not found, check base image and installed tools.

Do not fix build errors by copying random commands from internet. Understand which layer failed and why. A Docker build should be reproducible.

Common reasons for exiting: wrong command, missing environment variable, application crash, missing file, port already in use inside the container, permission error, or dependency unavailable. If a restart policy is set, Docker may restart it repeatedly.

Check exit code with inspect. Exit code 0 usually means the process finished normally. Non-zero means error. But the logs are usually the best first source.

Do not assume restart loops are Docker daemon problems. Most restart loops are application/configuration problems.

Check docker ps for port mapping. If no host port is shown, the host cannot access the app. Check app logs for the port it listens on. If the app listens on 127.0.0.1 inside the container, traffic from Docker port mapping may not reach it; use 0.0.0.0 for many web apps.

If port is correct, test curl from host. Then exec into the container and test from inside. This separates app issue from Docker networking issue.

Also check host firewall, VPN, corporate proxy, and port conflicts.

docker system df shows disk usage. docker container prune removes stopped containers. docker image prune removes dangling/unused images depending on flags. docker builder prune removes build cache. docker volume prune removes unused volumes and can delete important data if you are careless.

Never run prune commands blindly on a machine with important data. First inspect what exists. For learning machines, cleanup is common. For business or production systems, cleanup should be planned and documented.

Understand the difference between deleting containers and deleting volumes. Removing a Compose stack with down does not remove named volumes unless you add -v.

A production image should be built in CI, tagged clearly, scanned, and stored in a registry. Containers should run with least privilege. Secrets should come from a secret manager. Ports should be limited. Logs should go to a central system. Health checks should exist. Persistent data should have backups.

Resource limits protect the host from one container using all CPU or memory. Restart policies help recover from simple failures, but they do not replace monitoring. Rollback should be planned before deployment.

For larger systems, Docker images often run on orchestration platforms such as Kubernetes, ECS, Cloud Run, or Azure Container Apps. Docker skills still matter because the image is the package.

Docker is the base skill. It teaches image building and container running. Compose is the next step. It teaches how multiple services work together in one local stack. Kubernetes is orchestration. It runs containers across many machines, manages scaling, rollout, self-healing, service discovery, and configuration.

A practical learning path is Docker -> Compose -> Kubernetes. Many businesses use Compose for local development and Kubernetes or managed container platforms for production.

Do not compare them as if only one should exist. They solve different layers of the application lifecycle.

Your project folder should include app.py, requirements.txt, Dockerfile, .dockerignore, and README.md. The README should explain the business scenario, build command, run command, test command, stop command, and troubleshooting.

The app must listen on 0.0.0.0 so it can receive traffic through Docker port mapping. If it listens only on 127.0.0.1, your browser may not reach it.

The API service should use DB_HOST=db, not localhost. The database service should use a named volume for data. The Compose file should include environment variables. Only the API needs to publish a port for host access. The database can remain internal.

Your README should include run, stop, logs, reset database, common errors, and screenshots. This is a strong beginner-to-intermediate portfolio project.

Improve the Dockerfile with .dockerignore, trusted base image, non-root user, clean dependency installation, and no secrets. Scan the image using Docker Scout if available. Tag the image with a version and a commit-like tag. Write a README section explaining CI/CD steps even if you do not have a real pipeline.

The project should include screenshots or command outputs: build success, test success, scan command, image tag, and run result.

A good answer has four parts: definition, reason, example, and troubleshooting. For example, do not only say “volume stores data.” Say: A volume is Docker-managed persistent storage. It matters because container writable data can disappear after container removal. Example: PostgreSQL uses pgdata volume. Troubleshooting: inspect mounts and check the correct data path.

Use your projects as proof. Talk about your Python API, Compose app with database, and secure image CI/CD flow. Explain decisions like .dockerignore, non-root user, DB_HOST=db, named volumes, port mapping, and health checks.